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ABSTRACT 



The invention provides a method and system for duplicating 
all or part of a file system while maintaining consistent 
copies of the file system. The file server maintains a set of 
snapshots, each indicating a set of storage blocks making up 
a consistent copy of the file system as it was at a known time. 
Each snapshot can be used for a purpose other than main- 
taining the coherency of the file system, such as duplicating 
or transferring a backup copy of the file system to a 
destination storage medium. In a preferred embodiment, the 
snapshots can be manipulated to identify sets of storage 
blocks in the file system for incremental backup or copying, 
or to provide a file system backup that is both complete and 
relatively inexpensive. 
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At a step 372, the file server 110 reads the incremental 
mirror data structure into its file system and uses that data 
structure to update the base storage image 220. 

At a step 373, the file server 110 remounts the file system 
that was updated using the incremental mirror data structure. 5 

At a flow point 380, the file server 110 is ready to continue 
operations with the file system restored from the base 
storage image 220 and the incremental mirror data structure. 
Volume Replication 

At a flow point 350, the file server 110 is ready to perform 
a volume replication operation. 10 

At a step 351, the destination file system 120 initiates the 
volume replication operation. The destination file system 
120 sends an indicator of its newest master snapshot 210 to 
the file server 110, and requests the file server 110 to perform 
the volume replication operation. 15 

At a step 352, the file server 110 determines if it needs to 
perform a volume replication operation to synchronize with 
a second file server 110. In this case, the second file server 
110 takes the role of the destination file system 120, and 
initiates the volume replication operation with regard to the 20 
first file server 110. 

At a step 353, the file server 110 determines its newest 
master snapshot 210, and its master snapshot 210 corre- 
sponding to the master snapshot 210 indicated by the 
destination file system 120. 2 s 

If the file server 110 has at least one master snapshot 210 
older than the master snapshot 210 indicated by the 
destination file system 120, it selects the corresponding 
master snapshot 210 as the newest one of those. 

In this case, the method proceeds with the step 354. 30 

If the file server 110 does not have at least one master 
snapshot 210 older than the master snapshot 210 indicated 
by the destination file system 120 (or if the destination file 
system 120 did not indicate any master snapshot 210), it 
does not select any master snapshot 210 as a corresponding 35 
master snapshot. 

In this case, the method proceeds with the step 355. 

At a step 354, the file server 110 performs an incremental 
volume copying operation, responsive to the incremental 
difference between the selected corresponding master snap- 40 
shot 210, and the newest master snapshot 210 it has avail- 
able. The method 300 proceeds with the flow point 360. 

At a step 355, the file server 110 performs a full volume 
copying operation, responsive to the newest master snapshot 
210 it has available. The method 300 proceeds with the flow 45 
point 360. 

At a flow point 360, the file server 110 has completed the 
volume replication operation. The destination file system 
120 updates its master snapshot 210 to correspond to the 
master snapshot 210 that was used to make the file system 50 
transfer from the file server 110. 
Technical Appendix 

A technical appendix, tided " WAFL Image Transfer," and 
having the inventors named as authors, forms a part of this 
specification, and is hereby incorporated by reference as if 55 
fully set forth herein. 
Alternative Embodiments 

Although preferred embodiments are disclosed herein, 
many variations are possible which remain within the 
concept, scope, and spirit of the invention, and these varia- 60 
tions would become clear to those skilled in the art after 
perusal of this application. 

What is claimed is: 

1. A method for identifying storage blocks in a file system 
having a plurality of storage blocks, comprising the steps of: 65 
identifying a first storage image indicating a first set of 
member storage blocks selected from the plurality, the 



,118 B2 

18 

first storage image including a first bit plane indicating 
which member storage blocks are in the first storage 
image; 

identifying a second storage image indicating a second set 
of member storage blocks selected from the plurality, 
the second storage image including a second bit plane 
indicating which member storage blocks are in the 
second storage image; 

performing a logical operation on the first bit plane and 
the second bit plane to determine a logical difference 
between the first storage image and the second storage 
image; 

wherein at least the first storage image or the second 
storage image indicates member storage blocks form- 
ing a consistent file system other than an active file 
system. 

2. A method for identifying storage blocks in a file system 
having a plurality of storage blocks, comprising the steps of: 

identifying a first storage image indicating a first set of 
member storage blocks selected from the plurality, the 
first storage image including a first bit plane indicating 
which member storage blocks are in the first storage 
image; 

identifying a second storage image indicating a second set 
of member storage blocks selected from the plurality, 
the second storage image including a second bit plane 
indicating which member storage blocks are in the 
second storage image; 

performing a logical operation on the first bit plane and 
the second bit plane to determine a logical sum of the 
first storage image and the second storage image; 

wherein at least the first storage image or the second 
storage image indicates member storage blocks form- 
ing a consistent file system other than an active file 
system. 

3. A memory storing information including instructions, 
the instructions executable by a processor to identify storage 
blocks in a file system having a plurality of storage blocks, 
the instructions comprising: 

identifying a first storage image indicating a first set of 
member storage blocks selected from the plurality, the 
first storage image including a first bit plane indicating 
which member storage blocks are in the first storage 
image; 

identifying a second storage image indicating a second set 
of member storage blocks selected from the plurality, 
the second storage image including a second bit plane 
indicating which member storage blocks are in the 
second storage image; 

performing a logical operation on the first bit plane and 
the second bit plane to determine a logical difference 
between the first storage image and the second storage 
image; 

wherein at least the first storage image or the second 
storage image indicates member storage blocks form- 
ing a consistent file system other than an active file 
system. 

4. A memory storing information including instructions, 
the instructions executable by a processor to identify storage 
blocks in a file system having a plurality of storage blocks, 
the instructions comprising: 

identifying a first storage image indicating a first set of 
member storage blocks selected from the plurality, the 
first storage image including a first bit plane indicating 
which member storage blocks are in the first storage 
image; 
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identifying a second storage image indicating a second set in the file system, the instructions comprising: (a) 

of member storage blocks selected from the plurality, identifying a first storage image indicating a first set of 

the second storage image including a second bit plane member storage blocks selected from the plurality, the 

indicating which member storage blocks are in the nrst stor age image including a first bit plane indicating 

second storage image; 5 which mem b er storage blocks are in the first blocks 

performing a logical operation on the first bit plane and selected from the plurality, the second storage image 

the second bit plane to determine a logical sum of the including a second bit plane indicating which member 

first storage image and the second storage image; storage blocks are in the second storage image, (c) 

wherein at least the first storage image or the second performing a logical operation on the first bit plane and 

storage image indicates member storage blocks form- 10 the second bit plane to determine a logical sum of the 

ing a consistent file system other than an active file first storage image and the second storage image, 

svstem - wherein at least the first storage image or the second 

5. An apparatus including: storage image indicates member storage blocks form _ 

a storage medium that stores a file system having a 15 ing a consistent file system other than an active file 

plurality of storage blocks; system, 

a processor that executes instructions; and 7. A method as in claim 1, wherein the first bit plane and 

a memory that stores the instructions, the instructions tne second bit plane each form a column of a collection of 

executable by the processor to identify storage blocks bit planes, each row in the collection representing a storage 

in the file system, the instructions comprising: (a) 20 block. 

identifying a first storage image indicating a first set of 8. A method as in claim 2, wherein the first bit plane and 

member storage blocks selected from the plurality, the the second bit plane each form a column of a collection of 

first storage image including a first bit plane indicating bit planes, each row in the collection representing a storage 

which member storage blocks are in the first storage block. 

image, (b) identifying a second storage image indicat- 25 9. A memory as in claim 3, wherein the first bit plane and 

ing a second set of member storage blocks selected the bit plane each form a of a coUection of 

from the plurality, the second storage image including bit planes> each row in lhe coUection representing a storage 

a second bit plane indicating which member storage block 

blocks are in the second storage image, (c) performing 10. Amemory as in claim 4, wherein the first bit plane and 

a logical operation on the first bit plane and the second 30 , he ^ w ^ eacb form p 

bit plane to determine a logical difference between the .... / . „ . . 

first storage image and the second storage image, J p ' anes ' each row lD the coUectlon re P re **""g a storage 

wherein at least the first storage image or the second * . 

storage image indicates member storage blocks form- 1L M a PP aratus * in claim 5 > wherein the first bit plane 

ing a consistent file system other than an active file 35 and l . he second blt P lane each form a column of a coUection 

system. °^ bit planes, each row in the collection representing a 

6, An apparatus including: stora S e block - 

a storage medium that stores a file system having a f* " ^T* ^^T ?* ^ ^ 

plurality of storage blocks; *f** ^ blt P lane each l° rm a n Column of a coUectloa 

° ^ ot bit planes, each row in the collection representing a 

a processor that executes instructions; and storage block. 

a memory that stores the instructions, the instructions 

executable by the processor to identify storage blocks ***** 
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(57) ABSTRACT 

The present invention provides a method for keeping a file 
system in a consistent state and for creating read-only copies 
of a file system. Changes to the file system are tightly 
controlled. The file system progresses from one self- 
consistent state to another self -consistent state. The set of 
self-consistent blocks on disk that is rooted by the root inode 
is referred to as a consistency point. To implement consis- 
tency points, new data is written to unallocated blocks on 
disk. A new consistency point occurs when the fsinfo block 
is updated by writing a new root inode for the inode file into 
it. Thus, as long as the root inode is not updated, the state of 
the file system represented on disk does not change. The 
present invention also creates snapshots that are read-only 
copies of the file system. A snapshot uses no disk space when 
it is initially created. It is designed so that many different 
snapshots can be created for the same file system. Unlike 
prior art file systems that create a clone by duplicating the 
entire inode file and all of the indirect blocks, the present 
invention duplicates only the inode that describes the inode 
file. A multi-bit free-block map file is used to prevent data 
from being overwritten on disk. 

8 Claims, 40 Drawing Sheets 
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In step 760, entries 2326A-2326L in block 2326 of the 
blkmap file 2344 are updated as illustrated in FIG. 21E. The 
diagram illustrates that the snapshot 2 bit (BIT2) is updated 
as well as the FS-BIT and CP-BIT for each entry 
2326A-2326L. Thus, blocks 2304, 2308-2312, 2316-2318, 5 
2322, and 2326 are contained in snapshot 2 whereas blocks 
2306, 2314, 2320, and 2324 are not. In step 770; the dirty 
blocks 2308 and 2326 are written to disk. 

Further processing of snapshot 2 is identical to that for 
generation of a consistency point illustrated in FIG. 5. In 10 
step 540, the two fcinfo blocks are flushed to disk. Thus, 
FIG. 21F represents the WAFL file system in a consistent 
state after this step. Files 2340, 2342, 2344, and 2346 of the 
consistent file system, after step 540 is completed, are 
indicated within dotted lines in FIG. 21F. In step 550, the 15 
consistency point is completed by processing inodes that 
were not in the consistency point. 

Access Time Overwrites 

Unix file systems must maintain an "access time" (atime) 20 
in each inode. Atime indicates the last time that the file was 
read. It is updated every time the file is accessed. 
Consequently, when a file is read the block that contains the 
inode in the inode file is rewritten to update the inode. This 
could be disadvantageous for creating snapshots because, as 25 
a consequence, reading a file could potentially use up disk 
space. Further, reading all the files in the file system could 
cause the entire inode file to be duplicated. The present 
invention solves this problem. 

Because of atime, a read could potentially consume disk 30 
space since modifying an inode causes a new block for the 
inode file to written on disk. Further, a read operation could 
potentially fail if a file system is full which is an abnormal 
condition for a file system to have occur. 

In general, data on disk is not overwritten in the WAFL 
file system so as to protect data stored on disk. The only 
exception to this rule is atime overwrites for an inode as 
illustrated in FIGS. 23A-23B. When an "atime overwrites" 
occurs, the only data that is modified in a block of the inode 4Q 
file is the atime of one or more of the inodes it contains and 
the block is rewritten in the same location. This is the only 
exception in the WAFL system; otherwise new data is 
always written to new disk locations. 

In FIG. 23A, the atimes 2423 and 2433 of an inode 2422 45 
in an old WAFL inode file block 2420 and the snapshot inode 
2432 that references block 2420 are illustrated. Inode 2422 
of block 2420 references direct block 2410. The atime 2423 
of inode 2422 is "April 30 9:15 PM" whereas the atime 2433 
of snapshot inode 2432 is "May 1 10:00 AM". FIG. 23A 50 
illustrates the file system before direct buffer 2410 is 
accessed. 

FIG. 23B illustrates the inode 2422 of direct block 2410 
after direct block 2410 has been accessed. As shown in the 
diagram, the access time 2423 of inode 2422 is overwritten 55 
with the access time 2433 of snapshot 2432 that references 
it. Thus, the access time 2423 of inode 2422 for direct block 
2410 is "May 1 11:23 AM". 

Allowing inode file blocks to be overwritten with new 
atimes produces a slight inconsistency in the snapshot. The 60 
atime of a file in a snapshot can actually be later than the 
time that the snapshot was created. In order to prevent users 
from detecting this inconsistency, WAFL adjusts the atime of 
all files in a snapshot to the time when the snapshot was 
actually created instead of the time a file was last accessed. 65 
This snapshot lime is stored in the inode that describes the 
snapshot as a whole. Thus, when accessed via the snapshot, 
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the access time 2423 for inode 2422 is always reported as 
"May 1 10:00 AM". This occurs both before the update 
when it may be expected to be "April 30 9: 15 PM", and after 
the update when it may be expected to be "May 1 11:23 
AM". When accessed through the active file system, the 
times are reported as "April 30 9:15 PM" and "May 1 11:23 
AM" before and after the update, respectively. 

In this manner, a method is disclosed for maintaining a file 
system in a consistent state and for creating read-only copies 
of the file system. 

What is claimed is: 

1. A method for recording a plurality of data about a 
plurality of blocks of data stored in storage means compris- 
ing the steps of: 

maintaining a means for recording multiple usage bits per 
block of said storage means, wherein one bit of said 
multiple bits per block for each of said blocks indicates 
a block's membership in an active file system and one 
or more bits indicated membership in one or more 
read-only copies of a file system; and 

storing, in said means for recording multiple usage bits 
per block, multiple bits for each of said plurality of said 
blocks of said storage means. 

2. A method for recording a plurality of data about a 
plurality of blocks of data stored in storage means compris- 
ing the steps of: 

maintaining a means for recording multiple usage bits per 
block of said storage means, wherein one bit of said 
multiple bits per block for each of said blocks indicates 
a block's membership in an active file system and one 
or more bits indicated membership in one or more 
read-only copies of a file system; and 

storing, in said means for recording multiple usage bits 
per block, multiple bits for each of said plurality of said 
blocks of said storage means, at least one of said 
multiple bits being indicative of block reusability. 

3. An apparatus having at least one processor and at least 
one memory coupled to said at least one processor for 
recording a plurality of data about a plurality of blocks of 
data stored in storage means, said apparatus includes: 

a recording mechanism configured to record multiple 
usage bits per block of said storage means, responsive 
to said plurality of data about said plurality of said 
blocks of said storage means, at least one of said 
multiple bits being indicative of block reusability. 

4. An apparatus having at least one processor and at least 
one memory coupled to said at least one processor for 
recording a plurality of data about a plurality of blocks of 
data stored in storage means, said apparatus includes: 

a recording mechanism configured to record multiple 
usage bits per block of said storage means, responsive 
to said plurality of data about said plurality of said 
blocks of said storage means, wherein one bit of said 
multiple bits per block for each of said blocks indicates 
a block's membership in an active file system and one 
or more bits indicates membership in one or more 
read-only copies of a file system. 

5. A computer program product including: 

a computer usable storage medium having computer 
readable code embodied therein for causing a computer 
to record a plurality of data about a plurality of blocks 
of data stored in storage means, said computer readable 
code includes: 

computer readable program code configured to cause said 
computer to effect a recording mechanism to record 
multiple usage bits per block of said storage means, 
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responsive to said plurality of data about said plurality 
of said blocks of said storage means, at least one of said 
multiple bits being indicative of block reusability. 

6. An. A computer program product including: 

a computer usable storage medium having computer 5 
readable code embodied therein for causing a computer 
to record a plurality of data about a plurality of blocks 
of data stored in storage means, said computer readable 
code includes: 

computer readable program code configured to cause said 30 
computer to effect a recording mechanism to record 
multiple usage bits per block of said storage means, 
responsive to said plurality of data about said plurality 
of said blocks of said storage means, wherein one bit of 
said multiple bits per block for each of said blocks 15 
indicates a block's membership in an active file system 
and one or more bits indicates membership in one or 
more read-only copies of a file system. 

7. A computer program product including: 

a computer data signal embodied in a carrier wave having 
computer readable code embodied therein for causing a 
computer to record a plurality of data about a plurality 
of blocks of data stored in storage means, said com- 
puter readable code includes: 
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computer readable program code configured to cause said 
computer to effect a recording mechanism to record 
multiple usage bits per block of said storage means, 
responsive to said plurality of data about said plurality 
of said blocks of said storage means, at least one of said 
multiple bits being indicative of block reusability. 

8. A computer program product including: 

a computer data signal embodied in a carrier wave having 
computer readable code embodied therein for causing a 
computer to record a plurality of data about a plurality 
of blocks of data stored in storage means, said com- 
puter readable code includes: 

computer readable program code configured to cause said 
computer to effect a recording mechanism to record 
multiple usage bits per block of said storage means, 
responsive to said plurality of data about said plurality 
of said blocks of said storage means, wherein one bit of 
said multiple bits per block for each of said blocks 
indicates a block's membership in an active file system 
and one or more bits indicates membership in one or 
more read-only copies of a file system. 
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(57) ABSTRACT 

The present invention provides a method for keeping a file 
system in a consistent state and for creating read-only copies 
of a file system. Changes to the file system are tightly 
controlled. The file system progresses from one self- 
consistent state to another self-consistent state. The set of 
self-consistent blocks on disk that is rooted by the root inode 
is referred to as a consistency point. To implement consis- 
tency points, new data is written to unallocated blocks on 
disk. A new consistency point occurs when the fsinfo block 
is updated by writing a new root inode for the inode file into 
it. Thus, as long as the root inode is not updated, the state of 
the file system represented on disk does not change. The 
present invention also creates snapshots that are read-only 
copies of the file system. A snapshot uses no disk space when 
it is initially created. It is designed so that many different 
snapshots can be created for the same file system. Unlike 
prior art file systems that create a clone by duplicating the 
entire inode file and all of the indirect blocks, the present 
invention duplicates only the inode that describes the inode 
file. A multi-bit free-block map file is used to prevent data 
from being overwritten on disk. 

22 Claims, 40 Drawing Sheets 
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space. Further, reading all the files in the file system could 
cause the entire inode file to be duplicated. The present 
invention solves this problem. 

Because of a time, a read could potentially consume disk 
space since modifying an inode causes a new block for the 5 
inode file to written on disk. Further, a read operation could 
potentially fail if a file system is full which is an abnormal 
condition for a file system to have occur. 

In general, data on disk is not overwritten in the WAFL 
file system so as to protect data stored on disk. The only 
exception to this rule is a time overwrites for an inode as 10 
illustrated in FIGS. 23A-238. When an "a time overwrites" 
occurs, the only data that is modified in a block of the inode 
file is the a time of one or more of the inodes it contains and 
the block is rewritten in the same location. This is the only 
exception in the WAFL system; otherwise new data is is 
always written to new disk locations. 

In FIG. 23 A, the a times 2423 and 2433 of an inode 2422 
in an old WAFL inode file block 2420 and the snapshot inode 
2432 that references block 2420 are illustrated. Inode 2422 
of block 2420 references direct block 2410. The a time 2423 
of inode 2422 is "4/30 9: 15 PM" whereas the a time 2433 of 
snapshot inode 2432 is "5/1 10.00 AM". FIG. 23A illustrates 
the file system before direct buffer 2410 is accessed. 

FIG. 23B illustrates the inode 2422 of direct block 2410 
after direct block 2410 has been accessed. As shown in the 
diagram, the access time 2423 of inode 2422 is overwritten 25 
with the access time 2433 of snapshot 2432 that references 
it. Thus, the access time 2423 of inode 2422 for direct block 
2410 is "5/1 11:23 AM". 

Allowing inode file blocks to be overwritten with new a 
times produces a slight inconsistency in the snapshot. The a 30 
time of a file in a snapshot can actually be later than the time 
that the snapshot was created. In order to prevent users from 
detecting this inconsistency, WAFL adjusts the a time of all 
files in a snapshot to the time when the snapshot was actually 
created instead of the time a file was last accessed. This 35 
snapshot time is stored in the inode that describes the 
snapshot as a whole. Thus, when accessed via the snapshot, 
the access time 2423 for inode 2422 is always reported as 
"5/1 10:00AM 5 '. This occurs both before the update when it 
may be expected to be "4/30 9:15PM", and after the update 
when it may be expected to be "5/1 11:23AM". When 
accessed through the active file system, the times are 
reported as "4/30 9:15PM" and "5/1 11:23AM" before and 
after the update, respectively. 

In this manner, a method is disclosed for maintaining a file 
system in a consistent state and for creating read-only copies 45 
of the file system. 

What is claimed is: 

1. A method for recording a plurality of data about a 
plurality of blocks of data stored in a storage system, 
comprising the step of maintaining multiple usage bits for 50 
each of said plurality of blocks; 

wherein one bit of said multiple usage bits for each of said 
plurality of blocks indicates a block's membership in 
an active file system and plural bits of said multiple 
usage bits for each of said plurality of blocks indicate 55 
membership in plural read-only copies of a file system; 
and 

wherein said multiple usage bits for each of said plurality 
of blocks are maintained in said active file system. 

2. A method as in claim 1, wherein one or more bits of said 60 
multiple usage bits for each of said plurality of blocks 
further indicate block reusability. 

3. A method for generating a consistency point for a 
storage system, comprising the steps of: 

marking a plurality of inodes pointing to a plurality of 65 
modified blocks in a file system stored on said storage 
system as being in a consistency point; 
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flushing regular files to said storage system; 
flushing special files to said storage system; 
flushing at least one block of file system information to 

said storage system; and 
queuing dirty inodes after said step of marking and before 

said step of flushing at least one block of file system 

information; 

wherein said step of flushing said special files to said 
storage system further comprises the steps of: 
pre-flushing an inode for a blockmap file to an inode 
file; 

allocating space on said storage system for all dirty 
blocks in said inode and said blockmap files; 

flushing said inode for said blockmap file again 

updating a plurality of entries in said blockmap file 
wherein each entry of said plurality of entries rep- 
resents a block in said storage system; and 

writing all dirty blocks in said blockmap file and said 
inode file to said storage system. 

4. A method of maintaining data in a storage system, 
comprising the steps of: 

maintaining a root inode and inodes for a file system in the 
storage system, the root inode pointing directly or 
indirectly to the inodes, and each inode storing file data, 
pointing to one or more blocks in the storage system 
that store file data, or pointing to other inodes, with 
different levels of indirection possible for the inodes to 
point to the blocks; and 

maintaining an inode map and a block map for the file; 

wherein when said inodes are maintained in the storage 
system, each inode only points to blocks in the storage 
system having a same level of indirection as other 
blocks pointed to by that inode. 

5. A method as in claim 4, further comprising the step of 
creating a snapshot of the file system by copying the root 
inode. 

6. A method as in claim 5, wherein the block map 
indicates membership of blocks in one or more snapshots. 

7. A method as in claim 5, further comprising the step of 
deleting a snapshot from the storage system, wherein blocks 
that are only part of the deleted snapshot are released for 
re-use by the storage system. 

8. A memory storing information including instructions, 
the instructions executable by a processor to record a 
plurality of data about a plurality of blocks of data stored in 
a storage system, the instructions comprising the step of 
maintaining multiple usage bits for each of said plurality of 
blocks; 

wherein one bit of said multiple usage bits for each of said 
plurality of blocks indicates a block's membership in 
an active file system and plural bits of said multiple 
usage bits for each of said plurality of blocks indicate 
membership in plural read-only copies of a file system; 
and 

wherein said multiple usage bits for each of said plurality 
of blocks are maintained in said active file system. 

9. A memory as in claim 8, wherein one or more bits of 
said multiple usage bits for each of said plurality of blocks 
further indicate block reusability. 

10. A memory storing information including instructions, 
the instructions executable by a processor to generate a 
consistency point for a storage system, the instructions 
comprising the steps of: 

marking a plurality of inodes pointing to a plurality of 
modified blocks in a file system stored on said storage 
system as being in a consistency point; 

flushing regular files to said storage system; 
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flushing special files to said storage system; 

flushing at least one block of file system information to 

said storage system; sand 
queuing dirty inodes after said step of marking and before 

said step of flushing at least one block of file system 5 

information; 

wherein said step of flushing said special files to said 
storage system further comprises the steps of: 
pre -flushing, an inode for a blockmap file to an inode 

file; 10 
allocating space on said storage system for all dirty 

blocks in said mode and said blockmap files; 
flushing said inode for said blockmap file again; 
updating a plurality of entries in said blockmap file 

wherein each entry of said plurality of entries rep- 15 

resents a block in said storage system; and 
writing all dirty blocks in said blockmap file and said 

inode file to said storage system. 

11 . A memory storing information including instructions, 
the instructions executable by a processor to maintain data 20 
in a storage system, the instructions comprising the steps of: 

maintaining a root inode mode and inodes for a file system 
in the storage system, the root inode pointing directly 
or indirectly to the inodes, and each inode storing file 
data, pointing to one or more blocks in the storage 25 
system that store file data, or pointing to other inodes, 
with different levels of indirection possible for the 
anodes to point to the blocks; and 

maintaining an inode map and a block map for the file 
system; 30 

wherein when said inodes are maintained in the storage 
system, each inode only points to blocks in the storage 
system having a same level of indirection as other 
blocks pointed to by that inode. 

12. A memory as in claim 11, wherein the instructions 
further comprise the step of creating a snapshot or the file 
system by copying the root inode. 

13. A memory as in claim 12, wherein the block map 
indicates membership of blocks in one or more snapshots. 

14. A memory as in claim 12, wherein the instructions 
further comprise the step of deleting a snapshot from the 
storage system, wherein blocks that are only part of the 
deleted snapshot are released for re -use by the storage 
system. 

15. A system comprising: 
a processor; 

a storage system; and 

a memory storing information including instructions, the 
instructions executable by the processor to record a 
plurality of data about a plurality of blocks of data 50 
stored in the storage system, the instructions compris- 
ing step of maintaining multiple usage bits for each of 
said plurality of blocks; 

wherein one bit of said multiple usage bits for each of said 
plurality of blocks indicates a block's membership in 55 
an active file system and plural bits of said multiple 
usage bits for each of said plurality of blocks indicate 
membership in plural read-only copies of a file system; 
and 

wherein said multiple usage bits for each of said plurality 60 
of blocks are maintained in said active file system. 

16. A system as in claim 15, wherein one or more bits of 
said multiple usage bits for each of said plurality of blocks 
further indicate block reusability. 

17. A system comprising: ^ 
a processor; 

a storage system; and 
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a memory storing information including instruction, the 
instructions executable by the processor to generate a 
consistency point for the storage system, the instruc- 
tions comprising the steps of: (a) marking a plurality of 
inodes pointing to a plurality of modified blocks in a 
file system stored on said storage system as being in a 
consistency point; (b) flushing regular files to said 
storage system; (c) flushing special files to said storage 
system; (d) flushing at least one block of file system 
information to said storage system; and (e) queuing 
dirty inodes after said step of marking and before said 
step of flushing at last one block of file system infor- 
mation; and 

wherein said step of flushing said special files to said 
storage system further comprises the steps of: (f) pre- 
flushing an inode for a blockmap file to an inode file; 
(g) allocating space on said storage system for all dirty 
blocks in said inode and said blockmap files; (h) 
flushing said inode for said blockmap file again; (i) 
updating a plurality of entries in said blockmap file 
wherein each entry of said plurality of entries repre- 
sents a block in said storage system; and (j) writing all 
dirty blocks in said blockmap file and said inode file to 
said storage system. 

18. A system comprising: 
a processor; 

a storage system; and 

a memory storing information including instructions, the 
instructions executable by the processor to maintain 
data in the storage system, the instructions comprising 
the steps of: (a) maintaining a root inode and inodes for 
a file system in the storage system, the root inode 
pointing directly or indirectly to the inodes, and each 
inode storing file data, pointing to one or more blocks 
in the storage system that store file data, or pointing to 
other inodes, with different levels of indirection pos- 
sible for the inodes to point to the blocks; and (b) 
maintaining an inode map and a block map for the file 
system; 

wherein when said inodes are maintained in the storage 
system, each inode only points to blocks in the storage 
system having a same level of indirection as other 
blocks pointed to by that inode. 

19. A system as in claim 18, wherein the instructions 
further comprise the step of creating a snapshot of the file 
system by copying the root inode. 

20. A system as in claim 19, wherein the block map 
indicates membership of blocks in one or more snapshots. 

21. A system as in claim 19, wherein the instructions 
further comprise the step of deleting a snapshot from the 
storage system, wherein blocks that are only part of the 
deleted snapshot are released for re-use by the storage 
system. 

22. A system for maintaining data in storage means, 
comprising: 

means for maintaining a root inode and inodes for a file 
system in the storage systems, the root inode pointing 
to inodes, and each inode storing file data, pointing to 
one or more blocks in the storage means that store file 
data, or pointing to other inodes; and 

means for maintaining an inode map and a block map for 
the file system; 

wherein when said inodes are maintained in the storage 
system, each inode only points to blocks having a same 
level of indirection as other blocks pointed to by that 
inode. 
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ABSTRACT 



A method is disclosed for maintaining consistent states of a 
file system. The file system progresses from one self- 
consistent state to another self -consistent state. The set of 
self-consistent blocks on disk that is rooted by a root inode 
is referred to as a consistency point. The root inode is stored 
in a file system information structure. To implement con- 
sistency points, new data is written to unallocated blocks on 
disk. A new consistency point occurs when the file system 
information structure is updated by writing a new root inode 
into it. Thus, as long as the root inode is not updated, the 
state of the file system represented on disk does not change. 
The method also creates snapshots that are user-accessible 
read-only copies of the file system. A snapshot uses no disk 
space when it is initially created. It is designed so that many 
different snapshots can be created for the same file system. 
Unlike prior art file systems that create a done by duplicating 
an entire inode file and all indirect blocks, the method of the 
present invention duplicates only the inode that describes the 
inode file. A multi-bit free -block map file is used to prevent 
data referenced by snapshots from being overwritten on 
disk. 

20 Claims, 39 Drawing Sheets 
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In step 760, entries 2326A-2326L in block 2326 of the 
blkmap file 2344 are updated as illustrated in FIG. 21E. The 
diagram illustrates that the snapshot 2 bit (BIT2) is updated 
as well as the FS-BIT and CP-BIT for each entry 
2326A-2326L. Thus, blocks 2304, 2308-2312, 2316-2318, 
2322, and 2326 are contained in snapshot 2 whereas blocks 
2306, 2314, 2320, and 2324 are not. In step 770, the dirty 
blocks 2308 and 2326 are written to disk, as indicated in 
FIG. 21F. 

Further processing of snapshot 2 is identical to that for 
generation of a consistency point illustrated in FIG. 5. In 
step 540, the two fcinfo blocks are flushed to disk. Thus, 
FIG. 21F represents the WAFL file system in a consistent 
state after this step. Files 2340, 2342, 2344, and 2346 of the 
consistent file system, after step 540 is completed, are 
indicated within dotted lines in FIG. 21F. In step 550, the 
consistency point is completed by processing inodes that 
were not in the consistency point. 

Access Time Overwrites 

Unix file systems must maintain an "access time" (atime) 
in each inode. Atime indicates the last time that the file was 
read. It is updated every time the file is accessed. 
Consequently, when a file is read the block that contains the 
inode in the inode file is rewritten to update the inode. This 
could be disadvantageous for creating snapshots because, as 
a consequence, reading a file could potentially use up disk 
space. Further, reading all the files in the file system could 
cause the entire inode file to be duplicated. The present 
invention solves this problem. 

Because of atime, a read could potentially consume disk 
space since modifying an inode causes a new block for the 
inode file to be written on disk. Further, a read operation 
could potentially fail if a file system is full which is an 
abnormal condition for a file system to have occur. 

In general, data on disk is not overwritten in the WAFL 
file system so as to protect data stored on disk. The only 
exception to this rule is atime overwrites for an inode as 
illustrated in FIGS. 23A-23B. When an "atime overwrites" 
occurs, the only data that is modified in a block of the inode 
file is the atime of one or more of the inodes it contains and 
the block is rewritten in the same location. This is the only 
exception in the WAFL system; otherwise new data is 
always written to new disk locations. 

In FIG. 23A, the atimes 2423 and 2433 of an inode 2422 
in an old WAFL inode file block 2420 and the snapshot inode 
2432 that references block 2420 are illustrated. Inode 2422 
of block 2420 references direct block 2410. The atime 2423 
of inode 2422 is "4/30 9:15 PM" whereas the atime 2433 of 
snapshot inode 2432 is "5/1 10:00 AM". FIG. 23A illustrates 
the file system before direct buffer 2410 is accessed. 

FIG. 23B illustrates the inode 2422 of direct block 2410 
after direct block 2410 has been accessed. As shown in the 
diagram, the access time 2423 of inode 2422 is overwritten 
with the access time at which direct block 2410 was 
accessed. Thus, the access time 2423 of inode 2422 for 
direct block 2410 is "5/1 11:23 AM". 

Allowing inode file blocks to be overwritten with new 
atimes produces a slight inconsistency in the snapshot. The 
atime of a file in a snapshot can actually be later than the 
time that the snapshot was created. In order to prevent users 
from detecting this inconsistency, WAFL adjusts the atime of 
all files in a snapshot to the time when the snapshot was 
actually created instead of the time a file was last accessed. 
This snapshot time is stored in the inode that describes the 
snapshot as a whole. Thus, when accessed via the snapshot, 
the access time 2423 for inode 2422 is always reported as 
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"5/1 10:00AM". This occurs both before the update when it 
may be expected to be "4/30 9:15PM", and after the update 
when it may be expected to be "5/1 11:23AM". When 
accessed through the active file system, the times are 

5 reported as "4/30 9:15PM" and "5/1 11:23AM" before and 
after the update, respectively. 

In this manner, a method is disclosed for maintaining a file 
system in a consistent state and for creating read-only copies 
of the file system. 

10 We claim: 

1. A method for recording a plurality of data about a 
plurality of blocks of data stored in storage means compris- 
ing the steps of: 

maintaining a means for recording multiple usage bits per 
15 block of said storage means, wherein one bit of said 
multiple bits per block for each of said blocks indicates 
a block's membership in an active file system and one 
or more bits indicated membership in one or more 
^ read-only copies of a file system; and 

storing in said means for recording multiple usage bits per 
block multiple bits for each of a plurality of said blocks 
of said storage means. 

2. A method for maintaining a file system stored in 
25 non-volatile storage means at successive consistency points 

said file system comprising blocks of data, said blocks of 
data comprising blocks of regular file data and blocks of 
meta-data file data referencing said blocks of data of said file 
system, said meta file data comprising a file system infor- 
3Q mation structure comprising data describing said file system 
at a first consistency point said computer system further 
comprising memory means, said method comprising the 
steps of: 

maintaining a plurality of modified blocks of regular file 
35 data and meta-data file data in said memory means, said 
modified blocks of data comprising blocks of data 
modified from said first consistency point; 

designating as dirty blocks of meta-data file data refer- 
encing said modified blocks of regular file data and 
40 meta-data file data, said dirty blocks of meta-data file 
data comprising blocks of meta-data file data to be 
included in a second consistency point; 

copying said modified blocks of regular file data refer- 
enced by said dirty blocks of meta-data file data to free 
45 blocks of said non-volatile storage means; 

copying blocks comprising said modified blocks of meta- 
data file data referenced by said dirty blocks of meta- 
data file data to free blocks of said non-volatile storage 
means; 

modifying a copy of said file system information structure 
maintained in said memory means to reference said 
dirty blocks of meta-data file data: 

copying said modified file system information structure to 
55 said non-volatile storage means. 

3. The method of claim 2 wherein said blocks of meta-file 
data comprise one or more blocks of inode file data and one 
or more blocks of blockmap file data and wherein said step 
of copying said modified blocks of meta-data file data to free 

60 blocks of said non-volatile storage means further comprises 
the steps of: 

copying an inode referencing one or more blocks of 
blockmap file data to a block of inode file data main- 
tained in said memory means; 
65 allocating free blocks of said non-volatile storage means 
for said block of inode file data and one or more 
modified blocks of blockmap file data: 
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updating said inode referencing said one or more blocks 
of blockmap file data to reference said one or more free 
blocks of said non-volatile storage means allocated to 
said one or more modified blocks of blockmap file data; 

copying said updated inode to said block of inode file 
data; 

updating said one or more blocks of blockmap file data; 

writing said updated one or more blocks of blockmap file 
data and said block of inode file data to said allocated 
free blocks of said non-volatile storage means. 

4. A method for maintaining a file system comprising 
blocks of data stored in blocks of a non-volatile storage 
means at successive consistency points comprising the steps 
of: 

storing a first file system information structure for a first 
consistency point in said non-volatile storage means, 
said first file system information structure comprising 
data describing a layout of said file system at said first 
consistency point of said file system; 

writing blocks of data of said file system that have been 
modified from said first consistency point as of the 
commencement of a second consistency point to free 
blocks of said non-volatile storage means; 

storing in said non-volatile storage means a second file 
system information structure for said second consis- 
tency point, said second file system information struc- 
ture comprising data describing a layout said file sys- 
tem at said second consistency point of said file system. 

5. The method of claim 4 wherein said step of storing said 
first file system information structure in said non-volatile 
storage means comprises the step of: 

storing first and second copies of said first file system 
information structure at first and second locations 
respectively of said non-volatile storage means; 
and wherein said step of storing said second file system 
information structure in said non-volatile storage means 
comprises the steps of: 

overwriting said first copy of said first file system infor- 
mation structure with a first copy of said second file 
system information structure; and 
overwriting said second copy of said first file system 
information structure with a second copy of said second 
file system information structure. 

6. The method of claim 5 wherein said first and second 
locations of said non-volatile storage means comprise fixed 
predetermined locations of said non-volatile storage means. 

7. The method of claim 5 wherein each copy of said file 
system information structure comprises means for determin- 
ing a most recent version of said file system information 
structure and means for determining validity of said file 
system information structure, further comprising the steps 
of: 

after a system failure, reading said first and second copies 
of said file system information structure from said first 
and second locations of said non-volatile storage 
means; 

determining a most recent valid file system information 
structure from said first and second copies of said file 
system information structure. 
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8. A method for creating a plurality of read-only copies of 
a file system stored in blocks of a non-volatile storage 
means, said file system comprising meta-data identifying 
blocks of said non -volatile storage means used by said file 

5 system, comprising the steps of: 

storing meta-data for successive states of said file system 

in said non-volatile storage means; 
making a copy of said meta-data at each of a plurality of 
10 said states of said file system; 

for each of said copies of said meta-data at a respective 
state of said file system, marking said blocks of said 
non-volatile storage means identified in said meta-data 
as comprising a respective read-only copy of said file 
15 system. 

9. The method of claim 8 wherein said step of marking 
said blocks comprising a respective read-only copy of said 
file system comprises placing an appropriate entry in a 

2q means for recording multiple usage bits per block of said 
non-volatile storage means. 

10. The method of claim 9 wherein said means for 
recording multiple usage bits per block of said non-volatile 
storage meaas comprises a blockmap comprising multiple 

25 bit entries for each block. 

11. The method of claim 8 wherein said meta-data com- 
prises pointers to a hierarchical tree of blocks comprising 
said file system. 

12. The method of claim 8 wherein said meta-data com- 
30 prises structures representing files of said file system. 

13. The method of claim 12 wherein said structures 
representing files of said file system comprise inodes. 

14. The method of claim 8 further comprising the step of: 
preventing overwriting of said blocks marked as belong- 

35 ing to a read-only copy of said file system. 

15. The method of claim 8 comprising the step of unmark- 
ing said blocks marked as belonging to a read only copy of 
said file system when said read only copy of said file system 

40 is no longer needed. 

16. The method of claim 8 wherein a plurality of said 
blocks marked as belonging to a read-only copy of said file 
system comprise data ancillary to said file system, said 
method further including the steps of: 

45 allowing said ancillary data to be overwritten; and 

otherwise preventing overwriting of said blocks marked 
as comprising a read only copy of said file system. 

17. The method of claim 16 wherein said ancillary data 
comprises access time data. 

50 18. The method of claim 8 wherein said meta-data com- 
prises a root structure referencing structures representing 
files of said file system, and wherein said copies of said 
meta-data comprise copies of said root structure. 

19. The method of claim 18 wherein said root structure 
55 comprises a root inode. 

20. The method of claim 8 further comprising the step of 
using one or more of said read-only copies of said file 
system to back-up said blocks comprising one or more 
consistency points of said file system. 

60 
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[57] ABSTRACT 

The present invention provides a method for keeping a file 
system in a consistent state and for creating read-only copies 
of a file system. Changes to the file system are tightly 
controlled. The file system progresses from one self- 
consistent state to another self -consistent state. The set of 
self-consistent blocks on disk that is rooted by the root inode 
is referred to as a consistency point. To implement consis- 
tency points, new data is written to unallocated blocks on 
disk. A new consistency point occurs when the fsinfo block 
is updated by writing a new root inode for the inode file into 
it. Thus, as long as the root inode is not updated, the state of 
the file system represented on disk does not change. The 
present invention also creates snapshots that are read-only 
copies of the file system. A snapshot uses no disk space when 
it is initially created. It is designed so that many different 
snapshots can be created for the same file system. Unlike 
prior art file systems that create a clone by duplicating the 
entire inode file and all of the indirect blocks, the present 
invention duplicates only the inode that describes the inode 
file. A multi-bit free-block map file is used to prevent data 
from being overwritten on disk. 
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In general, data on disk is not overwritten in the WAFL 
file system so as to protect data stored on disk. The only 
exception to this rule is atime overwrites for an inode as 
illustrated in FIGS. 23A-23B. When an "atime overwrites" 
occurs, the only data that is modified in a block of the inode 
file is the atime of one or more of the inodes it contains and 
the block is rewritten in the same location. This is the only 
exception in the WAFL system; otherwise new data is 
always written to new disk locations. 

In FIG. 23A, the atimes 2423 and 2433 of an inode 2422 
in an old WAFL inode file block 2420 and the snapshot inode 
2432 that references block 2420 are illustrated. Inode 2422 
of block 2420 references direct block 2410. The atime 2423 
of inode 2422 is "4/30 9: 15 PM" whereas the atime 2433 of 
snapshot inode 2432 is "5/1 10:00 AM". FIG. 23A illustrates 
the file system before direct buffer 2410 is accessed. 

FIG. 23B illustrates the inode 2422 of direct block 2410 
after direct block 2410 has been accessed. As shown in the 
diagram, the access time 2423 of inode 2422 is overwritten 
with the access time 2433 of snapshot 2432 that references 
it. Thus, the access time 2423 of inode 2422 for direct block 
2410 is "5/1 11:23 AM". 

Allowing inode file blocks to be overwritten with new 
atimes produces a slight inconsistency in the snapshot. The 
time of a file in a snapshot can actually be later than the time 
that the snapshot was created. In order to prevent users from 
detecting this inconsistency, WAFL adjusts the atime of all 
files in a snapshot to the time when the snapshot was actually 
created instead of the time a file was last accessed. This 
snapshot time is stored in the inode that describes the 
snapshot as a whole. Thus, when accessed via the snapshot, 
the access time 2423 for inode 2422 is always reported as 
"5/1 10:00AM". This occurs both before the update when it 
may be expected to be "4/30 9:15PM", and after the update 
when it may be expected to be "5/1 11:23AM". When 
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accessed through the active file system, the times are 
reported as "4/30 9:15PM" and "5/1 11:23AM" before and 
after the update, respectively. 

In this manner, a method is disclosed for maintaining a file 
system in a consistent state and for creating read-only copies 
of the file system. 

We claim: 

1. A method for generating a consistency point compris- 
10 ing the steps of: 

marking a plurality of inodes pointing to a plurality of 
modified blocks in a file system as being in a consis- 
tency point; 

^ flushing regular files to storage means; 

flushing special files to said storage means; 

flushing at least one block of file system information to 
said storage means; and, 

requeueing any dirty inodes that were not part of said 
20 consistency point. 

2. The method of claim 1 wherein said step of flushing 
said special files to said storage means further comprises the 
steps of: 

pre-flushing an inode for a blockmap file to an inode file; 
25 allocating space on said storage means for all dirty blocks 
in said inode and said blockmap files; 
flushing said inode for said blockmap file again; 
updating a plurality of entries in said blockmap file 
30 wherein each entry of said plurality of entries repre- 
sents a block on said storage means; and, 
writing all dirty blocks in said blockmap file and said 
inode file to said storage means. 

* * * * * 
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RECOVERABLE VIRTUAL MEMORY HAVING 
PERSISTANT OBJECTS 

The present application is a continuation-in-part of 5 
U.S. Ser. No. 774,828, titled RECOVERABLE VIR- 
TUAL MEMORY filed Sept. 11, 1985, by Satish 
Thatte. 

BACKGROUND AND SUMMARY OF THE in 
INVENTION 

This invention relates to digital computer systems, 
and specifically to a virtual memory having recovery 
capabilities. 

In the future, as users of state-of-the-art symbolic 15 
computing machines develop large-scale, knowledge- 
based applications, they are expected to encounter 
major problems arising out of storage management 
problems in supporting large and complex knowledge/- 
data bases. The word storage is used herein in a broad 20 
sense to encompass virtual memory, file systems and 
databases. The problems can be primarily attributed to 
the dichotomy by which today's computers, including 
state-of-the-art symbolic computers such as the Texas 
Instruments EXPLORER and the Symbolics 3670, 23 
manage storage along two entirely different organiza- 
tions. These organizations can be referred to as the 
computational storage and the long-term storage. 

In symbolic/artificial intelligence (AI) processing, a 
representation of knowledge is a combination of data 30 
structures and interpretive procedures that, if used in 
the right way in a program, will lead to "knowledge- 
able" behavior. The goals of AI systems can be de- 
scribed in terms of cognitive tasks like recognizing ob- 
jects, answering questions, and manipulating robotic 35 
devices. The most important consideration in formulat- 
ing a knowledge representation scheme is the eventual 
use of the knowledge. The actual use of the knowledge 
in symbolic/AI programs involves three stages: (1) 
acquiring more knowledge, (2) retrieving facts from the 40 
knowledge base relevant to the problem at hand, and (3) 
reasoning about these facts in search of solutions. A 
number of different knowledge representation schemes, 
such as state-space representation, logic, procedural 
representation, semantic nets, production systems, and 45 
frames, have been developed by the knowledge repre- 
sentation community. The choice of the knowledge 
representation scheme very much depends on the appli- 
cation requirements. 

No matter which knowledge representation scheme 50 
is used, at some sufficiently low level of representation 
the knowledge is represented by memory objects inter- 
connected by pointers. These objects exhibit a structure 
which is defined by the interconnection graph of point- 
ers connecting the objects. The structure of objects 55 
created and manipulated by symbolic AI applications is 
usually very rich and complex. Moreover, both the 
information in objects, as well as the structure of ob- 
jects, can undergo rapid changes. 

In symbolic computing, objects representing a 60 
knowledge base are created and manipulated in the 
computational storage. As its name implies, the compu- 
tational storage contains objects to be manipulated by 
the processor of a computer system. These objects can 
be numbers, strings, vectors, arrays, records, linked 65 
lists, instructions, procedures, etc. These objects, both 
small and large, are usually identified by names. The 
names of objects serve as convenient handles or point- 
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ers that can be passed as procedure parameters, re- 
turned as procedure results, and stored in other objects 
as components. The names of objects are typically im- 
plemented as their virtual addresses. Programmers cre- 
ate and manipulate objects by using programming lan- 
guages, such as LISP and PROLOG. 

Typically, the computational storage is implemented 
as virtual memory, which consists of a hierarchy of 
memories: a fast, small semiconductor main memory, 
backed up by a slow, large disk to support paging. Ob- 
jects in the computational storage are accessed very 
rapidly as the processor can directly access them by 
specifying their addresses (real or virtual), often at a 
speed that matches the basic processor cycle time. The 
information stored in these objects is also processed and 
manipulated very efficiently as it is stored in a format 
defined by the processor architecture, and can therefore 
be directly interpreted by the processor hardware or 
microcode. 

Often, the information stored in the computational 
storage has a very rich structure; i.e., objects in the 
computational storage are interconnected by a rich and 
complex structure of pointers to match the require- 
ments of applications at hand. The structure of these 
objects is often dynamic. However, objects in the com- 
putational storage do not exist beyond the life times of 
programs that create them. When a program terminates 
or a system shutdown, or crash occurs, these objects 
cease to exist. Therefore, they are called short-lived or 
transient objects. To make these objects survive beyond 
the life times of programs that created them, i.e., to 
make them long-lived or persistent, they must be moved 
to the other storage organization, i.e., the long-term 
storage. 

As its name implies, the long-term storage is used to 
keep information for long periods of time. It is typically 
implemented on a disk-resident file system. The disk file 
system is logically different from the paging disk of the 
computational storage, even though the physical disk 
media may be shared by both. Examples of information 
stored in the long-term storage are files, directories, 
libraries, and databases. The long-term storage retains 
information in a reliable fashion for long periods of 
time. In order to store information beyond the life time 
of a program that periods of time. In order to store 
information beyond the life time of a program that cre- 
ates it in the computational storage, the information 
needs to be first mapped into a representation expected 
by the long-term storage and then transferred to it for 
long-term retention using a file input/output (I/O) op- 
eration or a database operation. 

The types of objects supported by the long-term 
storage are very restrictive (essentially files, directories, 
relations, etc.), and may match with the data structure 
requirements of many applications. The representation 
of information in the long-term storage is quite "flat." 
For example, a file may consist of a sequential stream of 
bits or bytes, such as ASCII characters. Files or rela- 
tions usually can neither hold procedural objects nor 
pointers to other objects in the long-term storage. Infor- 
mation in these objects can neither be directly ad- 
dressed nor directly processed by the processor, be- 
cause its representation is not compatible with the pro- 
cessor architecture. The information can be processed 
only after it is mapped into a representation expected by 
the computational storage and then transferred to it for 
processing. The translation overhead in mapping these 
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objects to/from a collection of files is quite substantial, In large data-intensive programs there is usually a 

too. considerable amount of code, which has been estimated 

In addition to the time overhead for translation and to be as high as 30% of the total, concerned with trans- 
mapping of objects between the computational and ferring data between files or a database, and the compu- 
long-term storages, there is additional space overhead, 5 tational storage. Much space and time is wasted by code 
as the information is essentially duplicated in virtual to perform translations between the transient and persis- 
memory and the file system. There is an apparent para- tent object worlds, which has adverse performance 
dox in that the computational storage, usually imple- impact. This is unsatisfactory because the effort and 
mented as a virtual memory, hides the existence of the time required to develop and execute the translation 
paging disk store; on the other hand, the long-term 10 00(16 caT1 considerable, and also because the quality 
storage makes the existence of the disk explicit to the an ? reliability of the application programs may be im- 
programmer. Thus, the programmer is faced with a paired by the mapping. The storage dichotomy also 
nonuniform storage model, where differences in ad- gives rise to much duplication of efTort in the operating 
dressing, function, and retention characteristics be- system design and DBMS design, 
tween the computational and long-term storages are 15 These problems, created by the storage dichotomy, 
visible above the processor architecture level. are considerably further complicated for symbolic/AI 

Programming languages, such as FORTRAN, Pas- computing. Processes on current symbolic machines 

cal, LISP, and Prolog, strongly reflect the dichotomy in sharc a sm « lc address space: i.e., there is no per-process 

storage organization. The specification of these lan- addrcs * s P ace ; Moreover the address space is not seg- 

guages almost invariably assumes long-term storage 20 m ^ t( ;d but is a single, linear address space. Such a 

objects (files) to have entirely different characteristics modcl °* ^ computational storage allows easy, effi- 

from computational objects. As a result, these program- Clcnt and ? cxlbl f . shann * of . ^J** 1 am ° n « mu,ti P lc 

ming languages cannot directly process information in Presses. Any object can point to any other object by 

the long-term storage the way they can process infor- „ ^ply holding a pointer to that object (usually imple- 

mation in the computational storage. This dichotomy 25 ™ nt ?™ a T** ° f thC **** ^ P° ,nted 

propagates throughout the whole system and cannot be ^) Avb^nly complex structures of objects intercon- 

hidden from the user. It shows up in differences be- d by fff™ t can ** 5 rc * ted and ™!»P^ d - 

tween names used for programming language objects * U ^ We ? 1 stra f™? ° f ° b J«: ts * very im P°«™< 

and names used for files and databaL. „ f ° h ^ ,0p r integrated and I pow- 

-n. j- 1. # i u j« *»r 30 e " u * software development environments available on 

The dichotomy also shows up in a different set of these ^j^lic comouters 
languages that has evolved to process information in the Unfortunately, current symbolic computers make a 
tang, erm storage^ These languages include various distinction between the computational and long-term 
so-called command l^guages^ such as the UNIX shell ftQ similaf iQ |oda/$ conventional computers. In 
language and the IBM TSO Command Language, that 35 ^ com utcrs , maki a si , c object ^rsistent 
are responsible among other things for performing b movi it t0 a flle m is nQt mean £g ful; ^ 
opera ions on files. The other class of languages which objccts that can ^ rcached from an ob / ect b fo f lowin 
operate on persistent objects are various database Ian- a „ out . going powers also need to be made persistent as 
guages, such as Square, Sequel, and Quel These lan- a si , c entit ^ and all in<oming po int ers pointing to 
guages can define database objects, and ^perform quenes 40 the entity must ^ » pr0 pe r i y taken care of." Such an 
and updates on them. Typically, such languages are entitV( howe ver, can be very large and moving it to a 
often interpreted, and are restrictive and arcane in na- flle syslem would ^ a complicated ^ expensive oper- 
ture compared to the more familiar programming lan- adon ; Conversely, the reverse move from a file system 
guages, which also enjoy the efficiency of compiled to the computational storage would be equally as corn- 
execution over interpreted execution. 45 p ii ca ted and expensive. 

As a consequence, the programmer must be aware of Many current adva nced programming techniques, 
the nonuniform storage model, and must explicitly especially as practiced in the symbolic/AI community, 
move information among storage media, based on the do not distinguish between procedures and data; proce- 
addressmg mechanisms, functions and retention charac- dures m im ^ which are themselves active. As the 
tenstics desired. Another consequence is that the nonu- 50 body of information being dealt with grows and be- 
niform storage model is an obstacle to programming comes more active, it becomes critical that the program 
generality and modularity as it increases potential types environment, consisting of complex objects intercon- 
of interfaces among programs. The hodgepodge of nected with rich pointer structures, survives for long 
mode-dependent programming languages, such as com- periods of time. Mapping and moving of such rich envi- 
mand languages, programming languages, debugging 55 ronments into today's file system or database for long- 
languages, and editing languages, makes fast and effi- term retention would involve substantial translation 
cient interaction with the system difficult. overhead, both in space and time. 

The mapping between transient and persistent objects Thus, there is a substantial difference between the 

is usually done in part by the file system or the data base representations of objects in the computational and 

management system (DBMS) and in part by explicit 60 long-term storages for symbolic/AI applications. The 

user translation code which has to be written and in- richer the structure of computational objects, the 

eluded in each program. This task imposes both space greater the difference and the bigger the effort needed 

and time penalties, and degrades system performance. to perform translation between these two representa- 

Frequently the programmer is distracted from his task tions. Emerging symbolic and AI applications will em- 

by the difficulties of understanding the mapping and 65 ploy increasingly sophisticated and complex structures 

managing the additional burden of coping with two on a large number of objects on which retrievals, 

disparate worlds: the programming language world and queries, inferences, reasoning, deductions, and compu- 

the DBMS world. tations will be performed. As can be anticipated, the 
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overhead to map long-term objects into computational those items that have been stored in files or a DBMS are 

objects and vice-versa for large knowledge-intensive considered to be valid, 

applications could be substantial. A system which implements a large uniform memory 

The current approach tfken by many researchers to is especially vulnerable to system crashes. Because per- 

facilitate knowledge-based applications is based on con- 5 sistent objects are stored in the virtual memory, they 

necting a symbolic computer to a database machine. can be corrupted by the crash. The most recent version 

This approach is not based on persistent memory, as it of a particular persistent object may or may not be 

neither addresses the storage dichotomy issues nor deals stored on the paging disk. The current value of large 

with the lifetime or interchangeability of procedure and objects may be partially on disk, and partially in RAM. 
data issues. There will be a mismatch between the data 10 Thus, the values stored on disk cannot be relied on, and 

model requirements of symboIic/AI applications and cannot merely be used to reload and restart the system 

the rigid data models supported by database machines. after a crash. 

Therefore, such approach appears to be inadequate for Thus, if it is desired to restore a virtual memory after 

expert database systems. These reservations are shared a crash, prior art file and DBMS systems cannot be 
by other researchers in the field. 15 used. It is necessary to devise some mechanism for pre- 

The persistent memory approach is based on a funda- serving the state of the virtual memory, 
mentally different foundation. The literature on persis- It is an object of the present invention to provide a 
tent memory dates back to 1962, when Kilbura pro- virtual memory which can recover from hardware fail- 
posed single-level storage, in which all programs and ures and software errors. It is a further object to provide 
data are named in a single context. (T. Kilburn, "One 20 a virtual memory which can be restored to an earlier, 
Level Storage System", IRE Trans. Electronic Cora- valid state to minimize loss of work. It is another object 
put., vol. EC- II, no. 2, Apr. 1962) Saltzer proposed a to provide a means for taking regular checkpoints of the 
direct-access storage architecture, where there is only a virtual memory to preserve valid states which can be 
single context to bind and interpret all objects. (J. H. restored. It is another object to provide an improved 
Salzer, "Naming and Binding of Objects". In R. Bayer recoverable paging scheme for virtual memories, 
et al, editors, Operating Systems, An Advanced Course, In order to provide for system recovery in case of a 
p. 99, Springer-Verlag, New York, N.Y., 1978. power failure, hardware failure or software error, 

Traiger proposed mapping databases into virtual ad- checkpoints are periodically taken of the state of the 
dress space. (I. L. Traiger, "Virtual Memory Manage- 3Q system. These checkpoints are marked and stored on 
ment for Database Systems", ACM Operating Systems disk. Changes made between a checkpoint and the next 
Review, pp. 26-48. Oct. 1982.) It seems that the simple checkpoint are also stored and marked, but are dis- 
data modeling requirements of the FORTRAN and carded in the event of a system failure. When there is a 
COBOL worlds discouraged productization of these system failure, the system is rolled back to the check- 
proposals because they are much more difficult to im- 35 point state, and processing resumes in a normal manner, 
plement than the conventional virtual memory and Virtual memory pages which are updated after the most 
database systems. recent checkpoints are stored on disk as sibling pages. 

The MIT MULTICS system and the IBM System/38 An efficient state indicator mechanism is provided for 

have attempted to reduce the storage dichotomy. How- determining which sibling page is to be read from or 

ever, both have major shortcomings for symbolic com- ^ written to disk when the corresponding virtual page is 

puting: unlike LISP machines, each process has its own referenced. This state indicator mechanism indicates 

address space. All persistent information is in files. A which pages on disk are included in the checkpoint 

file mapped into the address space of a process cannot state, and which contain information modified since the 

hold a machine pointer to a file mapped in the address checkpoint. In normal operation, the most recent ver- 

space of a different process. Thus, sharing of informa- 45 sion is used, but when a system failure occurs, only the 

tion among different processes is more difficult than most recent version stored before the checkpoint is 

with LISP machines. Furthermore, there is no auto- used. 

matic garbage collection, which is essential for support- The novel features which characterize the present 

ing symbolic languages. invention are defined by the appended claims. The fore- 

Recently, many researchers have proposed imple- 50 going and other objects and advantages of the invention 

menting persistent objects on top of a file system pro- will hereinafter appear, and for purposes of illustration, 

vided by the host operating system. Though persistent but not limitation, two preferred embodiments are 

and transient objects still reside in two separate storage shown in the drawings, 
organizations, persistent objects can be of any general 

type, such as number, vector, array, record, or list, and 55 BRIEF DESCRIPTION OF THE DRAWINGS 

can be manipulated with a common programming lan- FIG. 1 is a block diagram of a prior art computer 

guage such as ALGOL or LISP. However, there is a memory system architecture; 

large overhead to access persistent objects because their FIG. 2 is a block diagram of a computer memory 

pointers must be dereferenced by software, taking sev- architecture for implementing a persistent memory; 

era! machine cycles. 60 FIG. 3 is a block diagram illustrating a uniform, per- 

Systems having two-level memory storage can easily sistent system; 

recover from a power failure, hardware failure, soft- FIG. 4 is a representation of sibling virtual pages as 

ware error, or the like, which can be considered as a stored on a paging disk according to a first preferred 

group as "system crashes". After a system crash, any embodiment; 

hardware problems are repaired and the software is 65 FIG. 5 illustrates how some sibling pages are updated 

reloaded from long-term storage. All data and proce- according to a first preferred embodiment; 

dures which were in the virtual memory at the time of FIG. 6 illustrates how other sibling pages are updated 

the crash are discarded, and the system is restarted, and according to a first preferred embodiment; 
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FIG. 7 illustrates how singleton pages are updated ory objects are implemented as virtual addresses of the 

according to a first preferred embodiment; target objects. 

FIG. 8 illustrates the process of recovering from a As shown in FIG. 3, there is a distinguished object in 

system crash; * the uniform memory abstraction, called the persistent 

FIG. 9 illustrates a hierarchy of memory abstractions; 5 root 110, which defines persistent objects, 

and The persistent root 110 is a distinguished object lo- 

FIG. 10 is a page table according to a second pre- cated at a fixed virtual address and disk location. All 

ferred embodiment. objects that are in the transitive closure of the persistent 

DESCRIPTION OF THE PREFERRED ^^^JS^^S^ f °' l0W - 

PMRnniMFWT 10 g pointers, are persistent. The persistent root survives 

tMDuuiMLw i S y Stem shutdowns or crashes. Typically, the persistent 

The present recovery system for a virtual memory root may contain a pointer to a table that points to other 

will be described in the context of its implementation tables or structures of persistent objects and so on. 

with a single-level persistent memory, which will be Thus, the persistent root anchors all persistent objects, 

described first. It will be apparent to those skilled in the 15 The persistence attribute of an object depends solely 

art that the present recoverable virtual memory can be on whether that object can be prevented from being 

used with other virtual memories, such as those used garbage collected even after the program that created it 

with present virtual memory systems. has terminated: this can be easily arranged by making 

ppwwrcw-r wcmadv cvcTCM that ob J ect a member of the set of objects in the transi- 

PEKSISTENT MEMORY SYSTEM 2Q tjvc dh%m of the root Persistence based 

FIG. 1 shows a prior art computer architecture hav- solely on the persistent root rather than the properties 

ing separate computational and long-term storages. A of the storage medium allows a complete separation of 

centra] processor 10 has access to computational stor- the persistence attribute of an object from its type or 

age 12 and long-term storage 14. Long-term storage 14 relationship with other objects. Numbers, characters, 

is for retaining files, databases, etc., and is usually imple- 25 lists, procedures, environments, etc., can be persistent 

mented as one or more disks backed up by magnetic objects while they exist in virtual memory, 

tape. Computational storage 12 is a virtual memory, Therefore, an invocation of a procedure as a persis- 

usually implemented as a fast semiconductor RAM tent object is as easy and efficient as its invocation as a 

memory 16 and a paging disk 18. The computational transient object. In fact, from the machine point of 

storage 12 appears to the central processor 10 as a very 30 view, transient and persistent objects are indistinguish- 

large RAM. Virtual memory addresses which are not able. From the user point of view, there is no need to 

actually in the semiconductor memory 16 are located on treat transient and persistent objects differently; all the 

the paging disk 18 and loaded into the semiconductor user needs to know is that to make an object persistent, 

memory 16 when they are referenced by the central it has to be in the transitive closure of the persistent 

processor 10. 35 root. 

FIG. 2 shows a computer system having an architec- The processor contains a number of "registers." 

ture according to the present invention. A central pro- (101-108 are shown) The processor can access a mem- 

cesser 20 (CPU) has access only to a single, uniform ory object, i.e., read and write its individual words, if 

memory 22. The memory 22 preferably consists of a any of its registers holds a pointer to the object. The 

very large virtual memory, having semiconductor 40 word register in this context is used in a generic sense; 

RAM 24 backed up by a paging disk 26. This virtual can it may be a hardware register or a scratch-pad memory 

be very similar to the virtual memory 12 of FIG. 1. The in the processor. These registers define the transient 

CPU 20 may be an existing system, such as an EX- root 109 of the memory system. They do not survive a 

PLORER symbolic processor from Texas Instruments. system shutdown or crash. All objects that are in the 

The virtual memory 22 appears to the CPU 20 as a 45 transitive closure of the transient root, but not in the 

uniform, or single-level, memory store with a linear transitive closure of the persistent root, are called tran- 

address space. sient. All the remaining objects are garbage and are 

The uniform memory abstraction defines the storage reclaimed by a garbage collector, 

system architecture necessary to implement a persistent FIG. 3 shows an example snapshot of the memory 

memory according to the present invention. The persis- 50 system and categorizes objects within it. The arrows 

tent memory system is based on the uniform memory between objects and from CPU registers to objects 

abstraction, in which a processor views memory as a set represent pointers. Pointers always refer to the begin- 

of variable-sized blocks, or objects, of memory inter- ning of the object pointed to. Thus, the four pointers 

connected by pointers. The memory system has a very pointing into object 124, for example, all have the same 

large address space to support large knowledge-based 55 value and point to the beginning of block 124. By deter- 

applications. The persistent memory system is expected mining the transient closure of the persistent root 110, 

to store persistent objects, including "files," which and the transient root 109, it is seen that objects 126, 129 

could be very large in number and size. Therefore, the and 130 are transient; objects 120, 121, 122, 123, 124, 

size of the underlying address space should be suffi- 125, and 127 are persistent; and objects 128 and 131 are 

ciently large to support a practical system. However, 60 garbage. 

the concept of persistent memory does not depend on Each memory object consists of one or more memory 

the actual size of the address space. words, or cells, which are stored in consecutive virtual 

As previously explained, all processes within a sym- addresses. The processor 20 can access a memory ob- 

bolic computer share the same single, linear address ject, i.e., read and write its individual words, if any of its 

space. This allows a rich, complex structure of objects 65 registers holds a pointer to the object. For example, one 

interrelated by pointers, to be created and manipulated. method of accessing individual cells is as follows. If 

The structure of memory objects interconnected by register 101 contains a pointer to a memory object 123, 

pointers forms a graph. Pointers interconnecting mem- then the processor 20 can read the third word of the 
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memory object 123 by executing a READ(l f 3) instruc- that attempts to perform operations that are undefined 

tion, where "I" specifies the processor register 101, and or illegal on a particular object type cause traps to ap- 

"3" specifies the third word of the memory object (123), propriate exception handling routines; for example, an 

pointed to by register 101. The contents of register 101 attempt to add an integer to a string object would cause 

are added to "3" to develop the virtual address of the 5 an exception. Each memory reference can be checked 

word to be read Similarly the processor 20 can write for bounds, i.e., "j" in a logical address (i, j) should not 

data in the ^urthword of the memory object 123 by exceed the size of the object pointed to by processor 

executing a WRITE(1, 4) data instruction. The proces- register "t " 

sor 20 can access memory objects only via logical ad- The nature of the memory system requires that it be 

dres^a logicaj address consists of a pair (i,j). where 10 garbage collected and be free from the so-called dan- 

£Ld ^SSc^^S^ " *" J g 10 ** ab,e t0 make P"S« in a finite 

^Tnntinn nf^r T^l * r amount of memory space. Without the reclamation and 

rt J^JE TL y a 3 ^ I" thC Umf ^ rm me T" reusc of memory space occupied by an object proven to 

eans. characters, string LISP CONS celK arrSi f ° m objects) the system would eventu. 

records, procedures, o^yu^n^^lJ^ f*™™ t0 , a « nndin 8 « rt ™ out of -J™* 

level ob£cts can be implemented using one or more ^ bagC w " TCtI0n preferably done automatically tn 

memory objects interconnected by pointers. Applica- 20 re ^* m ^^ preferably as aprocess executing concur- 

tion-level objects are constructed by combining lan- f"^. iwer processes. This is not necessary to the 

guage-level objects invention, however, and garbage collection can occur 

The persistence property of objects is based solely on r*riods of machine non-use, such as overnight, 

whether or not an object is within the transitive closure The dangling reference problem arises if the memory 

of the persistent root 110. The persistence attribute of an 25 SpaC€ for . an " phc ! tly deleted ob ^ ect is reclaim ed with- 

object is a fundamental notion. It should depend only on ° 1 ut P' ovin 8 that thcre *" no outstanding pointers to 
whether the object can survive beyond the life time of tha ? ? bjecL If the s P axx o^P"** °Y a deleted object is 

a program that creates it It should neither depend on ^claimed prior to such a proof, then the outstanding 

the type of the object nor on the properties of the stor- renters to the object may point to empty space, i.e., 

age medium on which the object resides. Since there 30 una]located memory, or to some undesired object if the 

will usually be several sets of unrelated groups of ob- reclaimed space has been later allocated to the new 

jects which are persistent, the persistent root 110 will object In either case, the memory system integrity 

usually first point to an object which contains nothing would be violated. 

more than pointers to persistent objects. Any object, ^h e proof that there are no outstanding pointers to a 

transient or persistent, can point to any other object to 35 deleted object is embedded within the garbage collec- 

facilitate the unrestricted sharing desired in many sym- tor * A deleted object is specially marked as tombstoned 

bolic/AI computations. when it is deleted. All of the outstanding pointers to a 

In contrast to this mechanism of achieving persis- tombstoned block are detected by the garbage collector 

tence of objects based solely on the persistent root, in and tnen destroyed. When no pointers to a tombstoned 

today's machines, both conventional and symbolic, an 40 block are left, its space is automatically reclaimed, 

object becomes persistent only when it is stored in the Th c notion of the integrity of the memory system 

long-term storage, i.e., disk store. Even in MULTICS refers to the integrity of the structure of the graph of 

or IBM-System/38, only certain types of objects, i.e., memory objects interconnected by pointers. Under this 

files, can become persistent, while other types of ob- notion, the integrity of the memory system is preserved 

jects, such as procedures, cannot 45 because it is impossible to forge a pointer, no informa- 

With the persistent root 110, the persistence attribute tion outside the bounds of an object can be accessed, 

of an object solely depends on whether that object can undefined or illegal operations for any object type cause 

be prevented from being garbage collected even when appropriate exception handling actions, and there is no 

the program that created it has terminated; this can be dangling reference problem. Therefore, the graph of 

easily arranged by making that object a member of the 50 memory objects undergoes transition only from one 

set of objects in the transitive closure of the persistent consistent state to another, ensuring its structural integ- 

root 110. rity. 

The transience/persistence attribute of objects is not The uniform memory abstraction achieves its goal of 

necessarily a permanent attribute. An object may be being able to manipulate both transient and persistent 

created as a transient object, then it can become a persis- 55 objects with equal ease, flexibility and efficiency be- 

tent object solely on the basis of being in the transitive cause words of a memory object pointed to by a proces- 

closure of the persistent root 110, and then can revert sor register can be accessed with equal ease and speed, 

back to the transient state by getting out of the transitive independent of whether the object is transient or persis- 

closure of the persistent root 110, and so on. tent. Therefore, an invocation of a procedure as a per- 

Each pointer, implemented as a virtual address, is 60 sistent object is as easy and efficient as its invocation as 
tagged as a pointer within memory. This tagging mech- a transient object. In fact, as far as the machine is con- 
anism is used to ensure that the processor cannot spec- cerned, there is no distinction in processing transient 
ify, fabricate, or forge a pointer. The processor is al- and persistent objects. The uniform memory abstraction 
lowed to access memory only by reference to logical allows the same set of flexible and powerful operations 
memory blocks. There may be additional tagging infor- 65 on transient and persistent objects from a single pro- 
nation associated with each object to indicate its type, gramming language. New procedures can be invoked as 
such as integer, floating point number, string, array, list persistent objects or transient objects. None of the prior 
or closure. This tagging information is used to ensure art systems can do this; in those systems, access to per- 
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sistent objects is considerably slower or more compli- checkpoint, as described below, in this small time per- 

cated than access to transient objects. iod is negligibly small. 

The relationship between the parts of the overall When a system crash occurs, the present state of the 

memory system, including* the uniform and persistent entire uniform memory is assumed to be corrupted and 

memory, just described, and the recoverable virtual 5 no further computational progress is possible. When a 

memory, described below, is illustrated in FIG. 9. The system crash is detected the diagnostics routines are 

various levels of viewing the system are shown as a run, which attempt to identify if the failure occurred 

hierarchy of memory abstractions. due to a permanent hardware failure. If a permanent 

At the lowest level is the actual physical memory; the hardware failure is detected, then the faulty hardware, 
main semiconductor RAM memory and the paging 10 such as a processor board or a memory module, is re- 
disk. The next level of abstraction is the recoverable placed or repaired before any further recovery steps are 
virtual memory as described below. The next level taken. If no permanent hardware problem is detected, 
shows garbage collection of the virtual memory, which the recovery scheme rolls back the state of the entire 
can be done by one of several known methods. The memory system to the last checkpoint state stored on 
abstraction of uniform memory is one of several that 15 disk, as explained below. From the checkpoint state, 
can be implemented on top of a recoverable virtual normal computation can be resumed, 
memory. The persistent memory described above is It is expected that system crashes arising from soft- 
implemented as the next layer of abstraction, and the ware failures will be much more common than those 
concept of resilient objects, as described below is imple- arising from hardware failures. The average rate of 
mented on the persistent memory. Finally, user applica- 20 system crashes in a single-user machine is expected to be 
tions, such as expert database systems, are implemented on the order of a few crashes per month, and an average 
at the top of the hierarchy. recovery time of several minutes is acceptable (assum- 

RECOVERABLE VIRTUAL MEMORY ^^^."^1; 

25 The media failures that affect disk storage can arise 

The recoverable virtual memory will now be de- from disk head crashes, deterioration of the magnetic 
scribed. In order to preserve the contents of the persis- media itself, or bugs in the disk driver software. It is 
tent memory system in the event of a system crash, a expected that media failures are much less frequent than 
recovery capability at the level of the virtual memory system crashes. The average media failure rate may be 
itself is required. In today's machines, both conven- 3Q a few failures per year, and the average recovery time 
tional and symbolic, recovery capability at the level of after a media failure is expected to be a few hours, 
virtual memory is not required. If there is a system The reason that media failures are treated differently 
crash, information in the computational storage is as- from system crashes is that a media failure also corrupts 
sumed to be lost; but information in the long-term stor- the last checkpoint state on disk, and the roll-back tech- 
age survives because it resides on a non-volatile disk 33 nique used for recovery after a system crash would not 
store, which cannot be directly accessed by the proces- work. To deal with media failures, the last checkpoint 
sor. Applications or users are expected to transfer infor- state on disk needs to be archived on another media, 
mation from the computational storage to the long-term such as sa streaming tape. After a media failure, a failed 
storage for survival from crashes. After a system crash, disk needs to be replaced with a new one, which is then 
the system is restarted using the information found in 4Q initialized from the last archived checkpoint state, 
the long-term storage. Therefore, recovery is not a A preferred recovery scheme for recovering from 
major concern in today's machines. Such a luxury has to system crashes, as defined above, will now be described, 
be sacrificed in the persistent memory system, as there is The recovery scheme is based on a novel mechanism to 
no distinction between the computational and long-term take periodic checkpoints of the entire machine state in 
storage. 45 an efficient fashion, possibly initiated by the user or an 

Failures that are expected to be dealt with by the application program. The checkpointing scheme keeps 

recovery system can be classified into two broad cate- the entire machine state valid, within the last few min- 

gories: system crashes and media failures. System utes, on disk. 

crashes may be caused by power failures, or software The preferred embodiment utilizes a paged virtual 

errors or hardware failures. Software errors may arise 50 memory. Memory is paged to and from the paging disk 

from bugs in the application or systems software, incor- in blocks of predetermined size, for example two thou- 

rect data or commands, etc. Hardware failures account sand words, regardless of the size of memory objects 

for intermittent and permanent failures in the processor, 120-131. Thus, several small objects may reside on a 

main memory, or I/O devices and their controllers. single page, and large objects may extend over multiple 

Media failures, which account for disk storage failures, 55 pages. 

have been deliberately excluded from hardware failures When a new virtual page is materialized at the re- 
fer reasons explained below. quest of the memory allocator, it is placed in a properly 
A system crash can be signaled by hardware self- initialized main memory 24, and is simultaneously mate- 
checking circuits, a power failure detector interrupt, or rialized on the paging disk 26. As explained below, a 
by the inability of software error handling routines to 60 virtual page is materialized on disk in either the sibling 
recover from a software error. Hardware self-checking form or the singleton form. In the preferred sibling 
circuits typically include parity error detectors on form, two adjacent disk blocks are allocated to a virtual 
memories and buses, fire code detectors for disks, self- page; in the singleton form, a single disk block is allo- 
checking checkers, watech-dog processors, etc. It is cated. If the page is unlikely to be modified in the future 
assumed that the latency of a system crash detection, 65 (for example, a page containing instructions), it is mate- 
i.e., the time period between the system state corruption rialized in the singleton form to save disk space. If it is 
because of a system crash and the actual detection of the expected to contain data, it is materialized in the sibling 
crash, is quite small so that the probability of taking a form. 
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As shown in FIG. 4, the two adjacent disk blocks the other sibling, here x, contains outdated information, 

allocated for a virtual page in the sibling form are called If one sibling lias a timestamp smaller than T c u and the 

siblings. The sibling pages are referred to as x and x\ A other sibling has a larger timestamp, the sibling with the 

singleton page is denoted 4s s. smaller timestamp belongs to the checkpoint state and 

As shown in FIG. 4, the page header is contiguous to 5 the sibling with the larger timestamp is outside the 

the page data on disk. As a page is fetched from the checkpoint state. Because of the way that timestamps 

paging disk 26 into main memory 24, the header infor- are initialized and updates, both siblings cannot have a 

mation is scattered to some other predetermined page in timestamp larger than T c a*. The timestamp is the only 

main memory 24. When the page is written to disk, its mechanism to distinguish between pages that belong to 

header information is gathered together with the data. 10 the last checkpoint state and pages that do not. 

Scattering refers to the act of separating the page On a page fault, if the missing page is in the sibling 

header from the data, and gather refers to the reunion of form, both sibling pages are read in a single disk access 

the header and data. This is done because the data por- to reduce access time. This is possible because they are 

tion of the page corresponds to a page in main memory, located adjacent to each other on disk. It is expected 

and the header is additional information. To perform 15 that the time to fetch two adjacent disk pages is only 

this operation efficiently, the disk controller should marginally larger than the time to fetch a single page, 

have the scatter-gather capability. The scatter-gather For the TI EXPLORER LISP machine, the average 

capability is necessary only for performance and not for additional time to fetch the second disk page is expected 

the correct functionality of the recoverable virtual to be less than approximately 2% of the total, 

memory. 20 If TS(x)<TS(x')<T c AJt, i.e., if both siblings have 

When a page is written to the paging disk 26, it is timestamps smaller than Tm, then the sibling with the 
stamped with the time of the disk write operation. In larger timestamp, i.e., x\ is kept in main memory, and 
FIG. 4, TS(x) indicates the timestamp of x and TS(x') the other sibling, i.e., x, is discarded. The situation is 
indicates the timestamp of x\ TS(s) denotes the times- analogous for TS(x')<TS(x)<T f ft*. The sibling kept 
tamp of (s). A convenient place to record the timestamp 25 belongs to the checkpoint state, while the discarded 
is the page header. Timstamps written in headers will sibling contains useless information. When this page is 
survive system crashes. Timestamps are derived from a later written to disk, it is written over the disk space of 
timer that runs reliably even in the presence of system the discarded sibling x: it cannot be written over the 
crashes. The granularity of timestamps need only be disk space of x\ else the checkpoint state would be 
moderately smaller than the time for a disk write opera- 30 destroyed. After writing again to disk, the timestamp 
tion because pages cannot be written faster than the disk relationship becomes TS(x')<T c a*<TS(x). This opera- 
write speed. With a 10 milliseconds granularity, a 64 bit tion is illustrated in FIG. 5, which assumes the most 
timer can generate unique timestamps for 5.8 billion recent checkpoint was taken at 4 p.m. x\ with times- 
years. Therefore, a 64-bit timestamp field in page head- tamp = 3 pm, is kept in main memory. This sibling, as 
ers is more than adequate. 35 modified, is written back to disk at 5 pm over the disk 

When a page is materialized in sibling form, its sib- space of x. 

lings are materialized on disk with timestamps of -1 If TS(x)<T cA *<TS(x'), i.e., T C hk falls between the 

and -2, indicating that both are yet to be written. Al- timestamps of two siblings, the sibling with the larger 

ternate schemes are of course possible. When a page is timestamp, i.e., x\ is kept in main memory, and the other 

materialized in singleton form, it is initialized on disk 40 sibling, i.e., x, is discarded from main memory. This case 

with a time stamp of - 1. All disk blocks that have been is illustrated in FIG. 6. The difference between FIG. 5 

modified since their initial materialization on disk will and FIG. 6 lies in the space on disk where the page is 

have unique timestamps. written back. The page is written over its own disk 

The time of the last checkpoint operation is denoted space in FIG. 6, i.e., over x\ because x' is not part of the 
as T C hk- It is stored in a reliable fashion at a known disk 45 last checkpoint state and can be written over, while x 
location. Updating of T c hk must be assured to be reli- belongs to the checkpoint state and must be preserved, 
able; any errors made in updating T c hk will probably On the other hand, in FIG. 5, x' is written over the disk 
destroy the checkpoint scheme. One method for ensur- space of the other sibling, i.e., over x, for the reasons 
ing the integrity of the value T c hk is to keep two identi- explained above. When the page is modified and written 
cal copies. When T c a* is to be updated, the first copy is 50 to disk, the timestamp relation is unchanged, 
changed, then checked to ensure a proper update. A For a singleton page, if TS(s)<T c aa, i.e., its times- 
system crash during this operation would not efTect the tamp is smaller than T<**, then it must be converted to 
second copy, so that the original value of T c hk is pre- a sibling form when it is modified, as shown in FIG. 7. 
served. After the first copy is updated, the second copy This conversion is necessary because the checkpointed 
is updated and checked. A system crash during this 55 state must not be overwritten. After the conversion, the 
operation will not efTect the system integrity, as the timestamp relationship becomes TS(x)<T c a*<TS(x'). 
proper new value of T c hk » in the first copy. Thus, one When a singleton is converted to a sibling, it is copied 
is assured of a proper value for T C hk at all times. to, say x, while its modified version is written over x\ 

A checkpoint state is "pristine" in the sense that it On the other hand, if the timestamp of a singleton page 

cannot be accessed during normal operation, and re- 60 is larger than T c aa, then its conversion to a sibling is not 

mains unperturbed unless there is a media crash. A required because it is not a part of the checkpoint state, 

singleton page with a timestamp smaller, or earlier, than Thus, each virtual page on disk is stored in either of 

Tchk belongs to the checkpoint state; oh the other hand, the two different forms: sibling or singleton, depending 

a singleton page with a timestamp larger, or later, than on whether the virtual page is update-active or update- 

Jchk, is outside the checkpoint state. For sibling pages, 65 inactive, respectively. As these names suggest, virtual 

if TS(x)<TS(x , )<T c w, i.e., if both siblings have times- pages that have been recently updated are termed up- 

tamps smaller than T c w, then the sibling with the larger date-active, and those that have not been recently up- 

timestamp, here x\ belongs to the checkpoint state, and dated are termed update-inactive. To reduce the disk 
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space requirement, a sibling page is converted back to log, and then all the pages can be actually written out. 
the singleton form when both siblings remain update- Alternatively, a small non- volatile random -access mem- 
inactive for a long period, defined by a threshold pa- ory can be used to buffer entries in the Expunge log. A 
rameter in the system. If TS(x)<TS(x')<T C Ajt and background process would then be responsible for emp- 
T C AA— TS(V)> (threshold), then the disk space manager 5 tying this buffer to disk. 

reclaims their space by converting them into a singleton It is highly desirable to reduce the time required to 

form. Singleton s contains the original contents and complete the checkpoint operation. The major part of 

timestamp of the more recent sibling. the checkpoint operation is the cleaning up of dirty 

If there is a system crash, when the page table is pages by writing them to disk. Dirty pages are those 

restored it will point to the old disk location of the 10 which have been modified since the last checkpoint was 

converted page. This old location may now contain taken. This time must be reduced so that there is no 

unrelated information, as it may have been reallocated user-perceived pause attributed to the checkpointing 

by the disk manager. Thus, it is necessary to keep a process. Such pauses would be irritating for interactive 

Forward-Page log (FP-log) outside of the virtual mem- users and disruptive for time-critical applications such 

ory address space. 13 as graphics and animation. 

Whenever a sibling is converted to a singleton, and The time for checkpointing can be reduced by means 
vice versa, an entry is written to the FP-log. This entry of two techniques. The first is by making sure that the 
identifies the virtual page, and shows the old (before fraction of dirty pages in main memory at the check- 
conversion) and new (after conversion) disk addresses. point time is reasonably small by continuously cleaning 
The FP-log allows the correct disk location of pages 20 up the dirty pages between successive checkpoints. This 
which have been converted to be recalculated after a can be done by a continuously operating background 
system crash, even though the information in the page process. A good heuristic to select a dirty page for 
table portion of the checkpoint state is incorrect. Pro- cleaning up is "select a dirty page that has not been 
cessing of the FP-log will be discussed further in con* modified recently'*. The second is by cleaning up the 
nection with FIG. 8, 25 dirty pages at the checkpointing time with no user-per- 

Whenever a sibling or singleton is modified, a sibling ceived pause, as explained below, 
is written back to disk at a time after the last checkpoint. The checkpoint process may be initiated by an appli- 
Since it is written after the last checkpoint but before a cation or the system. At checkpoint time, a checkpoint 
potential system crash, it might contain corrupted infor- process takes over. It saves all processor registers, the 
mation. If there is indeed a system crash, that sibling 30 transient root, into a snapshot object, which is part of 
must be expunged from the machine state by demoting the virtual memory. The snapshot object resides at a 
its timestamp to — 1 during the post-crash recovery. fixed and known disk location, and can be optionally 
When its timestamp is demoted to — K the sibling is made a part of the transitive closure of the persistent 
effectively rendered useless, and would be discarded if root. All dirty pages in the main memory are then writ- 
paged in during normal operation resumed after the 33 ten to disk. Finally, for implementations using times- 
post-crash recovery. If the timestamp were not de- tamps, T c hk is updated on disk to the current time, com- 
moted, the sibling would be retained in main memory, if pleting the checkpoint process. As described earlier, 
paged in, and the roll back process would fail to recre- T c hk must either be successfully updated, or the old 
ate the checkpoint state. value retained. Immediately after the checkpoint opera- 
Siblings whose timestamps must be demoted during 40 tion, TS(x) and TS(x') must both be less than T cA * for all 
the post-crash recovery can be located using a log, sibling pages, and TS(s) must be less than T c hk for all 
called the Expunge log. The Expunge log keeps track of singleton pages. In the alternative embodiment, all sib- 
a page which is written out to disk for the first time, if ling pages must be in either state Sib-1 or Sib-2. Option- 
and only if, the page is part of the last checkpoint and ally, all previously transient objects become persistent, 
the page has not yet been written to disk since the last 45 as they are now members of the transitive closure of the 
checkpoint. This occurs if TS(x)<TS(x ! )<T c *a or persistent root by virtue of the transient root becoming 
TS(x')<TS(x)<T f A&. The virtual page identification a persistent object. 

and its disk address, along with the time of its writing to A further reduction of the perceived time used to 
disk, are appended to the end of the Expunge log. The take a checkpoint can be made by utilizing a copy-on- 
Expunge log must be maintained outside the virtual SO write technique to clean up dirty pages in main mem- 
memory for it must survive system crashes. This can be ory. Using this technique, all dirty pages in main mem- 
done by writing it onto a different disk, for example, or ory are marked as copy-on-write; any attempt to write 
to a specially set aside sector of the paging disk. In any to them will cause a trap as explained below. This mark- 
event, the Expunge log is not addressable by the CPU. ing would generally be done in a page table, a data 

If a system crash occurs before the next checkpoint, 55 structure for keeping track of the location of virtual 
then as part of the post-crash recovery process, the memory pages. The processor registers are then re- 
Expunge log is processed: each page recorded in the log stored from the snapshot object and the normal opera- 
since the last checkpoint is located on disk, and the tion resumes. All these operations involve only one disk 
sibling with the higher timestamp is expunged by de- operation to store the processor registers into the snap- 
moting its timestamp to — 1. By demoting its timestamp 60 shot object on disk. Therefore, all these operations can 
to - 1, the sibling is rendered useless. be completed in well below 100 milliseconds on a typi- 

Maintenance of the Expunge log requires disk ac- cal Winchester disk system, allowing the resumption of 

cesses. This requirement can be amortized over a set of normal operation without any user-perceived pause, 

several pages. Instead of writing an entry to the Ex- The copy-on-write mechanism allows a very rapid 

punge log each time a page in the set is written out, a « resumption of the normal operation as no time is spent 

number of pages in the set can be queued up as they are in writing the dirty pages back to disk before the normal 

on their way out to disk. Entries for all pages in the set operation resumes. After normal operation resumes, 

can be first constructed and written into the Expunge any attempt by the processor to modify the pages 
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marked as copy-on-write causes a trap. The trap han- 
dler first makes a copy of the page in main memory and 
this new copy is modified, but the original copy is not 
disturbed as it belongs to (he checkpointed state. Any 
further attempts to modify are directed to this new 
copy. Until all pages marked as copy-on-write are 
cleaned up by writing them to disk, no newly created 
dirty pages are cleaned up. While the copy-on-write 
pages are being cleaned up, the normal operation con- 
tinues. Cleaning up of dirty pages is preferably accom- 
plished by a system routine running concurrently with 
user processes. Only when all pages marked as copy-on- 
write are cleaned up is the T e ftk variable updated to be 
the current time, signaling the completion of a check- 
pointing operation. Note that at the end of a check- 
pointing operation* the entire machine state is stored on 
disk. 

FIG. 8 indicates the activities during a post-crash 
recovery. After a system crash, diagnostics are run to 
detect any permanent hardware failures. If the diagnos- 
tics effort does not indicate a permanent hardware fail- 
ure, or after any indicated failure is repaired, the FP-Iog 
is processed. This involves reading each entry in the 
FP-Iog, and using the new disk location contained 
therein to update the page table as stored in the check- 
point state. Next, each page indicated by the Expunge 
log is located on disk, and the sibling with the higher 
timestamp is expunged by demoting its timestamp to 
— 1 as explained above. Singletons with timestamps 
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and Tchk- The four states of sibling pages are: Both 
siblings written before T c a*. with x being older than x' 
in state Sib-1 and x' older than x in state Sib-2, as illus- 
trated in FIG. 5: x written before the checkpoint and x' 
written after (state Sib-3) as shown in FIG. 6; and x' 
written before the checkpoint with x written after (state 
Sib-4), which is the converse of FIG. 6. These four 
states are set forth in Table I. 

TABLE I 



Sute 
Name 


3-bit 

Encoding 


Description 


bit position 


2 10 




Sib-1 


xOO 


both written before checkpoint. 






it' more recent 


Sib-2 


xO 1 


both written before checkpoint, 






x more recent 


Sib-3 


010 


x written before checkpoint 






x' written after checkpoint 


Sib-4 


01 1 


x' written before checkpoint 






x written after checkpoint 


Sing-1 


1 1 0 


s written before checkpoint 


Sing-2 


1 ] 1 


s written after checkpoint 



25 



The actual value of the timestamps used in the previ- 
ous embodiment is important only for the purposes of 
comparison with timestamps of siblings and the most 
recent checkpoint. The alternative embodiment simply 
abstracts the relationships between the timestamps into 



four states as described above. Only two bits are neces 

greater than T chk are also expunged. Then the state of 30 f* 1 ? [° ™ 1 ^*i t, J??J 1 ^ h .^???A h ?.. f ° U !.? a ! CS * S,b ; 
the machine is rolled back to the state corresponding to T ^ * u ~ * * 



the last checkpoint by restoring the processor registers 
from the snapshot object. 

The system must be reconciled with the external 
world prior to restart. Part of the restored checkpoint 
state is related to the system configuration and I/O 
interfaces specific to the checkpoint time. This state 
must now be reconciled with the current time and con- 
figuration. For example, if the checkpoint state contains 



35 



ling page. Table I also shows the two states which are 
necessary if singleton pages are to be used. This requires 
that the states be encoded using three bits. This is much 
smaller than the size of the timestamps associated with 
each page in main memory, and provides numerous 
advantages as will be shown below. 

As in standard virtual memory management mecha- 
nisms, a page table is used to locate pages on disk when 
a page fault occurs. The page table is a data structure 



a timer variable, this variable must be adjusted to reflect 40 which associates virtual page addresses with disk ad 



the current time. At this point the normal operation 
from the checkpoint state can be resumed. 

The problem of media or disk failures, as described 
earlier, can be addressed by archiving the last check- 
pointed state on disk to some other storage medium 45 
such as a streaming tape. This process is called the 
checkpoint archival process. If a disk failure occurs, a 
new disk can replace the damaged one, and the ar- 
chived state can be used to initialized the new disk. 
Only those pages of the last checkpointed state that are 50 
dirty with respect to the last archived state need be 
copied to the archival medium. Pages that are dirty 
with respect to the last archived state indicate same 
with a flag in the page header. This flag can also be 
maintained in the page table. The next checkpoint can- 55 
not be taken until an ongoing checkpoint archival pro- 
cess, if any, is completed. The checkpoint archival pro- 
cess is a relatively infrequent operation and is expected 
to occur at most a few times a week, perhaps as an 
overnight operation, or when no user is working on the 60 
machine. 

An alternative embodiment for selecting which sib- 
ling page to use from the disk works in a similar manner, 
but avoids the necessity for the use of page headers and 
the generation and maintenance of time stamps. The 65 
basis for operation of this mechanism is recognition of 
the fact that only four states are needed to describe the 
relationship between the timestamps of siblings x and x' 



dresses, and typically contains other status information. 
A simplified form of such a page table data structure is 
shown in FIG. 10. When a page fault occurs, the virtual 
page address is located in the page table, typically 
through hashing or other search techniques, and the 
associated physical disk address is retrieved. In the 
preferred embodiment, the bits necessary to describe 
the state condition of the siblings is contained in three 
state bits in the page table. The physical disk address 
information is always directed to one sibling, for exam- 
ple x, and the other sibling, x\ is always located at some 
disk position which is calculable from the physical disk 
address of x. For example, both siblings can be stored in 
adjacent blocks, or the same track/block for different 
disk heads, or in the same position on different disk 
drives. As an alternative, both physical disk addresses 
can be stored in the page table, so that sibling pages may 
be scattered randomly over the disk. This alternative 
method has the drawback of increasing the size of the 
page table. 

When a page fault occurs, instead of reading in both 
sibling pages and discarding one, as described in con- 
nection with the first embodiment, only the correct disk 
page is read in. The correct disk page address is calcu- 
lated by examining the state bits, and choosing the cor- 
rect sibling. The sibling which is read into memory, and 
which is written to for a page out operation, is the same 
as that described in the timestamp embodiment: only the 
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method of calculation for choosing the appropriate 
pages modified. 

When pages are written to disk, the relative time-or- 
write of the sibling pages and T c kk may change, in the 
manner illustrated in FIGS. 5 and 6. This means that the 5 
state bits will need to be modified during a page out 
operation. Referring to Table I, if state is Sib-1, both 
siblings were written before the last checkpoint, with x' 
more recent. Thus, x' is fetched into main memory. If x' 
is modified, it is written out to the disk space occupied 10 
by sibling x, with the state being changed to Sib-4. This 
preserves x\ which is part of the checkpointed state. 

An analogous situation occurs when a page is read in 
and state is Sib-2. When state is Sib-2, and the dirty page 
is written to disk, the state of the page changes to Sib-3. 15 

If a page is read in, and the state is Sib-3 or Sib-4, the 
state of the page remains unchanged when a dirty page 
is written out to disk. It will be recognized that these 
state changes correspond to preservation of the check- 
pointed state as described for the embodiment utilizing 20 
times tamps. 

Table I indicates states which are used for singleton 
pages (Sing-1 and Sing-2). In many cases, a system will 
use only sibling states, which would require only two 
state bits in the page table. If singleton pages, as de- 25 
scribed with the previous embodiment, are used, a third 
bit must be added and stored in the page table. The two 
possible states for a singleton page are: s written before 
or after the most recent checkpoint. These correspond 
to states Sing-1 and Sing-2 in Table I. 30 

If a page fault indicates a page is to be read in having 
state Sing-2, that page can simply be read into memory. 
It is not necessary to make a copy and convert the page 
into sibling form. When the page is written out again, 
the state remains Sing-2, and the page is written into its 35 
old disk location. 

If a page having a state Sing-1 is paged in, it is neces- 
sary to convert that page to sibling form. This is because 
the singleton page is part of the checkpointed state, and 
must be preserved. The conversion operation is the 40 
same as shown in FIG. 7. In that example, the state is 
changed from Sing-1 to Sib-4 as shown in Table I. 

When a checkpoint is taken, T e hk will be more recent 
than all pages which have been written to disk. There- 
fore, the states of some pages will need to be changed as 45 
part of the checkpoint operation. Once the checkpoint 
is completed, all pages will be in either Sib-1, Sib-2 or 
Sing-1 states. Siblings that were in state Sib-3 changed 
to Sib-1, and those who were in Sib-4 changed to Sib-2. 
Pages which were originally in Sib-1, Sib-2, or Sing-1, 50 
do not change state. 

If a page has been in state Sib-1 or Sib-2 for a long 
time, the older sibling contains information which has 
been obsolete for a long time. Such a page can be con- 
verted to the singleton form for the reasons described in 55 
connection with the embodiment using timestamps. The 
contents of the more recent sibling are copied to a sin- 
gleton page, S, which will be in state Sing-1. 

The bits marked *V* can be used for deciding when a 
conversion to the singleton form should take place. A 60 
procedure, which could be a background process, con- 
stantly scans all entries in the page table with a prese- 
lected frequency. This frequency is preferably quite 
low, and can be set as desired according to the system 
designed. This background process inspects the "x", or 65 
"idle bit", for all pages which are in the states Sib-1 or 
Sib-2. If the idle bit is "0", it is changed to "1" and the 
scan continues. If the idle bit is found to be a "1", that 
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page has not been updated since the previous scan 
through the page table, and that page is a candidate to 
be converted to singleton form. When a singleton page 
is converted to sibling form, or pages in states Sib-3 or 
Sib-4 are changed to the corresponding state Sib-1 or 
Sib-2 as result of a checkpoint being taken. The idle bit 
is set to zero. The length of time which an unmodified 
page will remain in sibling form before being converted 
into singleton form depends entirely on the scan rate of 
the scanning process. 

The advantages of the second embodiment over that 
of the first are several. It is not necessary to maintain a 
timestamp for each disk page, so that no page header is 
needed. Since there is no page header, there is no re- 
quirement that the disk controller be able to perform a 
scatter-gather operation as described earlier. Since only 
one sibling is read in, as opposed to reading two and 
then discarding one, it is not necessary that siblings be 
physically contiguous on the disk. They may be sepa- 
rated, and it is only necessary that the location of one 
sibling can be calculated from that of the other. Alterna- 
tively, the page table can be enlarged slightly to include 
both siblings in the page table. Since the simple case 
requires only two extra bits in the page table, or three if 
singleton pages are used, and a few extra bits are avail- 
able on many systems, the second embodiment can be 
implemented on any existing virtual memory system. 

Generally, pages in sibling form and singleton form 
will be stored in different locations on disk in order to 
optimize disk usage. A page table within the virtual 
memory address space indicates the disk location of 
each allocated virtual page. The page table is a persis- 
tent object, and is checkpointed as are all other objects. 
When a sibling is converted to a singleton, or vice versa, 
the page table is updated to point to the new disk loca- 
tion for that page, and the old location is deallocated 
and made available for later use. 

When the embodiment using state bits in the Page 
Table is implemented, it is not necessary to use the 
Expunge log. Pages written after the checkpoint in 
states Sib-3, Sib-4, and Sing-2 may be corrupted and 
should be ignored. Simply rolling the page table back to 
the checkpointed state restores the correct state as de- 
fined in Table I. No changing of timestamps or states is 
necessary. 

TECHNICAL ADVANTAGES 

A virtual memory using a state indicator in a page 
table can be made recoverable from system crashes. 
This can be done without placing any additional bur- 
den, such as a scatter-gather capability, on the disk 
controller. This allows for relatively easy implementa- 
tion of a persistent memory on nearly any existing vir- 
tual memory system. Some advantages of this system 
over one using timestamps include: no timestamps or 
page headers are necessary (which uses less memory on 
disk and in main memory), no expunge log is needed, 
and the overall implementation is realtively straightfor- 
ward. 

Although a preferred embodiment has been described 
in detail, it is understood that various substitutions, 
additions and modification may become apparent to 
those skilled in the art. These changes may be made 
without departing from the spirit and scope of the in- 
vention, which is defined by the appended claims. 

What is claimed is: 

1. A computer system having a recoverable virtual 
memory, comprising: 
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(a) a virtual memory having a random access memory 
and a backing store: 

(b) a plurality of pages stored in the backing store, 
each of said pages corresponding to a virtual mem- 
ory page; 5 

(c) a plurality of memory objects allocated within 
said virtual memory, said memory objects inter- 
connected by pointers; 

(d) a persistent root having a pointer to a memory io 
object, wherein all memory objects in the transitive 
closure of said persistent root define a set of long 
term objects not subject to reclamation by the sys- 
tem during garbage collection; 

(e) means for taking checkpoints of the current state 15 
of said virtual memory at selected times, said means 
including means for materializing a sibling page 
corresponding to each of a plurality of said virtual 
memory pages, wherein the latest written version 2 o 
of a virtual memory page and its respective sibling 
page which was written before a current check- 
point state is used to define the checkpoint state; 
and 

(f) a state indicator identified wiih each virtual mem- 25 
ory page and each sibling page for indicating 
which of said virtual memory or sibling pages 
stored in the backing store comprise a most recent 
checkpoint. 

2. The computer system of claim 1, wherein the com- 
puter system is reset to the most recent checkpoint state 
after a system failure. 

3. The computer system of claim 1, wherein the back- 
ing store is a paging disk. 35 

4. The computer system of claim 1, further compris- 
ing a page table for indicating the correspondence be- 
tween virtual memory pages and backing store pages, 
wherein said state indicators are located in said page 
table. 40 

5. The computer system of claim 4, wherein said state 
indicators comprise a plurality of n bits, whereby 2 n 
states can be defined by said state indicator. 

6. The computer system of claim 1, wherein sibling 45 
pages share a single state indicator with their associated 
disk pages. 
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7. The system of claim 1, wherein said persistent root 
is a memory object located within the virtual memory 
space. 

8. A method for recovering from a system failure in a 
virtual memory having a plurality of virtual pages 
stored on a backup store, comprising the steps of: 

(a) providing a persistent root having a pointer to an 
object in the virtual memory, wherein objects in 
the transitive closure of the persistent root define a 
set of nontransient objects which are not subject to 
reclamation by the system during garbage collec- 
tion; 

(b) identifying a state indicator with each virtual page 
when that page is written to the backup store; 

(c) taking periodic checkpoints of the virtual memory 
by storing the states of any registers and the persis- 
tent root onto the backup store at a predetermined 
location, and storing state indicators for the virtual 
pages at a predetermined location; 

(d) saving a current version of each virtual page in 
existence when the periodic checkpoint is taken; 
and 

(e) when a system failure occurs, restoring a check- 
point state of the virtual memory by discarding all 
virtual pages whose state indicator indicates that 
they were written to the backup store after the 
most recent checkpoint was taken. 

9. The computer system of claim 8, wherein step (c) 
includes the step of changing the state indicator for 
virtual pages containing the page table to indicate that it 
is a part of the checkpoint state. 

10. The computer system of claim 8, wherein step (d) 
comprises the steps of: 

(d.l) materializing sibling pages corresponding to 
each of said virtual pages which do not have re- 
spective sibling pages, and writing said sibling 
pages to the backing store when such virtual pages 
are updated, wherein the sibling pages contain 
updated versions of the virtual pages and the vir- 
tual pages are unchanged; and 

(d.2) for virtual pages already having respective sib- 
ling pages, leaving unchanged the virtual page or 
the sibling page which was most recently stored 
before the last checkpoint was taken, and storing 
an updated version of the virtual page in the re- 
maining of the virtual page or sibling page. 
***** 
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